-
Notifications
You must be signed in to change notification settings - Fork 4
Description
Run Information
| Architecture | x64 |
|---|---|
| OS | Windows 10.0.18362 |
| Baseline | 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4 |
| Compare | fe1607679b977b453d43c4de276968dd9ae6d5ec |
| Diff | Diff |
Improvements in System.Collections.CreateAddAndClear<Int32>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| SortedSet - Duration of single invocation | 51.94 μs | 45.68 μs | 0.88 | 0.01 | False | |||||
| ImmutableSortedSet - Duration of single invocation | 179.85 μs | 164.94 μs | 0.92 | 0.01 | False | |||||
| ImmutableList - Duration of single invocation | 101.71 μs | 89.08 μs | 0.88 | 0.02 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndClear<Int32>*'Payloads
Histogram
System.Collections.CreateAddAndClear<Int32>.SortedSet(Size: 512)
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 45.67613022941969 < 50.054110001727324.
IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 77.79091002341295 (T) = (0 -45967.448897016846) / Math.Sqrt((636384.9539038617 / (299)) + (92727.96894886105 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.11877436088007147 = (52163.086111434626 - 45967.448897016846) / 52163.086111434626 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.CreateAddAndClear<Int32>.ImmutableSortedSet(Size: 512)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 164.9414298245614 < 168.83490861742422.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 28.2660621643388 (T) = (0 -166082.7369351791) / Math.Sqrt((4395956.838199926 / (299)) + (2980543.748657064 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.0618742612189658 = (177036.75538311191 - 166082.7369351791) / 177036.75538311191 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.CreateAddAndClear<Int32>.ImmutableList(Size: 512)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 89.08496428571429 < 96.82860696067956.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 61.842070840397945 (T) = (0 -89169.62147396228) / Math.Sqrt((2307053.8824507645 / (299)) + (872604.1082623651 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.13116046066080025 = (102630.71307940323 - 89169.62147396228) / 102630.71307940323 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Architecture | x64 |
|---|---|
| OS | Windows 10.0.18362 |
| Baseline | 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4 |
| Compare | fe1607679b977b453d43c4de276968dd9ae6d5ec |
| Diff | Diff |
Improvements in System.Reflection.Attributes
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| IsDefinedClassHitInherit - Duration of single invocation | 475.86 ns | 438.57 ns | 0.92 | 0.07 | False | Trace | Trace | |||
| IsDefinedMethodBaseHitInherit - Duration of single invocation | 505.33 ns | 464.21 ns | 0.92 | 0.06 | False | |||||
| IsDefinedMethodOverrideMissInherit - Duration of single invocation | 501.27 ns | 469.46 ns | 0.94 | 0.08 | False | |||||
| IsDefinedClassHit - Duration of single invocation | 487.33 ns | 440.26 ns | 0.90 | 0.12 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Reflection.Attributes*'Payloads
Histogram
System.Reflection.Attributes.IsDefinedClassHitInherit
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 438.57259661841033 < 456.7786079876879.
IsChangePoint: Marked as a change because one of 1/9/2023 8:29:54 AM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 39.06867646681837 (T) = (0 -438.84170280776834) / Math.Sqrt((207.86481978433716 / (299)) + (15.803539921155485 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.09571512165522761 = (485.2914311815499 - 438.84170280776834) / 485.2914311815499 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```### Baseline Jit Disasm
```assembly
; System.Reflection.Attributes.IsDefinedClassHitInherit()
mov rcx,22ECAF79ED8
mov rdx,22ECAF79F00
mov r8d,1
jmp qword ptr [7FFA756193C8]; System.RuntimeType.IsDefined(System.Type, Boolean)
; Total bytes of code 32
; System.RuntimeType.IsDefined(System.Type, Boolean)
push rdi
push rsi
sub rsp,28
mov rsi,rcx
mov edi,r8d
test rdx,rdx
je short M01_L02
mov rcx,offset MT_System.RuntimeType
cmp [rdx],rcx
jne short M01_L01
M01_L00:
test rdx,rdx
je short M01_L03
mov r8,offset MT_System.RuntimeType
cmp [rdx],r8
jne short M01_L03
movzx r8d,dil
mov rcx,rsi
add rsp,28
pop rsi
pop rdi
jmp qword ptr [7FFA757EF000]; System.Reflection.CustomAttribute.IsDefined(System.RuntimeType, System.RuntimeType, Boolean)
M01_L01:
mov rcx,rdx
mov rax,[rdx]
mov rax,[rax+58]
call qword ptr [rax]
mov rdx,rax
jmp short M01_L00
M01_L02:
mov ecx,0E9
mov rdx,7FFA754F4000
call CORINFO_HELP_STRCNS
mov rcx,rax
call qword ptr [7FFA757E5BA0]
int 3
M01_L03:
mov rcx,offset MT_System.ArgumentException
call CORINFO_HELP_NEWSFAST
mov rsi,rax
call qword ptr [7FFA75785648]
mov rdi,rax
mov ecx,0E9
mov rdx,7FFA754F4000
call CORINFO_HELP_STRCNS
mov r8,rax
mov rdx,rdi
mov rcx,rsi
call qword ptr [7FFA756BDBD0]
mov rcx,rsi
call CORINFO_HELP_THROW
int 3
; Total bytes of code 189Compare Jit Disasm
; System.Reflection.Attributes.IsDefinedClassHitInherit()
mov rcx,21CB106B4F8
mov rdx,21CB106B520
mov r8d,1
jmp qword ptr [7FF8D19B93C8]; System.RuntimeType.IsDefined(System.Type, Boolean)
; Total bytes of code 32; System.RuntimeType.IsDefined(System.Type, Boolean)
push rdi
push rsi
sub rsp,28
mov rsi,rcx
mov edi,r8d
test rdx,rdx
je short M01_L02
mov rcx,offset MT_System.RuntimeType
cmp [rdx],rcx
jne short M01_L01
M01_L00:
test rdx,rdx
je short M01_L03
mov r8,offset MT_System.RuntimeType
cmp [rdx],r8
jne short M01_L03
movzx r8d,dil
mov rcx,rsi
add rsp,28
pop rsi
pop rdi
jmp qword ptr [7FF8D1B8F000]; System.Reflection.CustomAttribute.IsDefined(System.RuntimeType, System.RuntimeType, Boolean)
M01_L01:
mov rcx,rdx
mov rax,[rdx]
mov rax,[rax+58]
call qword ptr [rax]
mov rdx,rax
jmp short M01_L00
M01_L02:
mov ecx,0E9
mov rdx,7FF8D1894000
call CORINFO_HELP_STRCNS
mov rcx,rax
call qword ptr [7FF8D1B85BA0]
int 3
M01_L03:
mov rcx,offset MT_System.ArgumentException
call CORINFO_HELP_NEWSFAST
mov rsi,rax
call qword ptr [7FF8D1B25648]
mov rdi,rax
mov ecx,0E9
mov rdx,7FF8D1894000
call CORINFO_HELP_STRCNS
mov r8,rax
mov rdx,rdi
mov rcx,rsi
call qword ptr [7FF8D1A5DBD0]
mov rcx,rsi
call CORINFO_HELP_THROW
int 3
; Total bytes of code 189System.Reflection.Attributes.IsDefinedMethodBaseHitInherit
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 464.21334104088754 < 479.90053645031685.
IsChangePoint: Marked as a change because one of 1/9/2023 3:58:12 PM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 43.203907879421294 (T) = (0 -461.2785938687879) / Math.Sqrt((223.54645906472078 / (299)) + (18.74129844932296 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.10590987142709027 = (515.919569098757 - 461.2785938687879) / 515.919569098757 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Reflection.Attributes.IsDefinedMethodOverrideMissInherit
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 469.46392847473214 < 478.48772646293617.
IsChangePoint: Marked as a change because one of 1/9/2023 3:58:12 PM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 15.080878984945256 (T) = (0 -469.3222084596509) / Math.Sqrt((213.01727900607807 / (299)) + (182.41604652610516 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.08794220685143164 = (514.5750762563807 - 469.3222084596509) / 514.5750762563807 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Reflection.Attributes.IsDefinedClassHit
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 440.26146786817515 < 461.80375843954914.
IsChangePoint: Marked as a change because one of 1/8/2023 4:08:48 AM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 37.403131333265854 (T) = (0 -439.511091456915) / Math.Sqrt((216.90719751509855 / (299)) + (17.828124148878537 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.09540303567479137 = (485.8639911363972 - 439.511091456915) / 485.8639911363972 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Architecture | x64 |
|---|---|
| OS | Windows 10.0.18362 |
| Baseline | 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4 |
| Compare | fe1607679b977b453d43c4de276968dd9ae6d5ec |
| Diff | Diff |
Improvements in System.Collections.Sort<Int32>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| Array_Comparison - Duration of single invocation | 29.66 μs | 25.23 μs | 0.85 | 0.01 | False | |||||
| Array_ComparerClass - Duration of single invocation | 29.64 μs | 25.04 μs | 0.84 | 0.00 | False | |||||
| Array_ComparerStruct - Duration of single invocation | 35.00 μs | 31.33 μs | 0.90 | 0.01 | False | |||||
| LinqQuery - Duration of single invocation | 38.17 μs | 33.75 μs | 0.88 | 0.01 | False | |||||
| LinqOrderByExtension - Duration of single invocation | 38.33 μs | 34.67 μs | 0.90 | 0.01 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Sort<Int32>*'Payloads
Histogram
System.Collections.Sort<Int32>.Array_Comparison(Size: 512)
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 25.229884 < 28.35812238571429.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 173.16249586197134 (T) = (0 -25112.597280885777) / Math.Sqrt((56485.88749501119 / (299)) + (11228.252987832124 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.1542223824374112 = (29691.72600388353 - 25112.597280885777) / 29691.72600388353 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.Sort<Int32>.Array_ComparerClass(Size: 512)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 25.040995999999996 < 28.205588033333335.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 149.07867568162393 (T) = (0 -25081.66954845155) / Math.Sqrt((73843.08487620296 / (299)) + (16899.548970046642 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.15922131745241758 = (29831.47654559153 - 25081.66954845155) / 29831.47654559153 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.Sort<Int32>.Array_ComparerStruct(Size: 512)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 31.33234133333334 < 33.27548114358975.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 55.59867456333919 (T) = (0 -31305.1419062604) / Math.Sqrt((177432.9135300424 / (299)) + (103669.80595089601 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.11454730078167981 = (35354.95677397184 - 31305.1419062604) / 35354.95677397184 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.Sort<Int32>.LinqQuery(Size: 512)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 33.74641384615384 < 36.69364468571428.
IsChangePoint: Marked as a change because one of 1/6/2023 2:04:15 PM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 32.981985973214094 (T) = (0 -34905.49433166833) / Math.Sqrt((294584.7940115831 / (299)) + (178904.21963234092 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.08275603027196897 = (38054.75476935328 - 34905.49433166833) / 38054.75476935328 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.Sort<Int32>.LinqOrderByExtension(Size: 512)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 34.67483428571428 < 36.412250436813196.
IsChangePoint: Marked as a change because one of 1/6/2023 5:32:27 AM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 40.58332838879311 (T) = (0 -34934.77786147186) / Math.Sqrt((263286.858829293 / (299)) + (106125.97764062008 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.0806625813621342 = (37999.952088573635 - 34934.77786147186) / 37999.952088573635 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Architecture | x64 |
|---|---|
| OS | Windows 10.0.18362 |
| Baseline | 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4 |
| Compare | fe1607679b977b453d43c4de276968dd9ae6d5ec |
| Diff | Diff |
Improvements in System.Collections.ContainsTrueComparer<Int32>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| SortedSet - Duration of single invocation | 37.64 μs | 30.41 μs | 0.81 | 0.00 | False | |||||
| ImmutableSortedSet - Duration of single invocation | 37.72 μs | 33.34 μs | 0.88 | 0.00 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrueComparer<Int32>*'Payloads
Histogram
System.Collections.ContainsTrueComparer<Int32>.SortedSet(Size: 512)
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 30.410661003236243 < 35.26340905629143.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 58.92581861955677 (T) = (0 -30533.26449607834) / Math.Sqrt((197245.08827584476 / (299)) + (255394.99499278617 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.1761147263967729 = (37060.09255699208 - 30533.26449607834) / 37060.09255699208 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.ContainsTrueComparer<Int32>.ImmutableSortedSet(Size: 512)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 33.33773809523809 < 35.60359647509266.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 72.5578858344467 (T) = (0 -32742.939104785375) / Math.Sqrt((90310.0255664583 / (299)) + (84680.61765431883 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.12493697532826514 = (37417.80669691573 - 32742.939104785375) / 37417.80669691573 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Run Information
| Architecture | x64 |
|---|---|
| OS | Windows 10.0.18362 |
| Baseline | 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4 |
| Compare | fe1607679b977b453d43c4de276968dd9ae6d5ec |
| Diff | Diff |
Improvements in System.Collections.ContainsFalse<Int32>
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| SortedSet - Duration of single invocation | 41.03 μs | 37.16 μs | 0.91 | 0.01 | False | |||||
| ImmutableSortedSet - Duration of single invocation | 42.15 μs | 36.49 μs | 0.87 | 0.01 | False |
Repro
General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md
Payloads
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsFalse<Int32>*'Payloads
Histogram
System.Collections.ContainsFalse<Int32>.SortedSet(Size: 512)
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 37.15679056754596 < 39.07662562882765.
IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 41.36812197885888 (T) = (0 -36869.76000015153) / Math.Sqrt((752198.9934455368 / (299)) + (192308.76841529066 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.10638003772565663 = (41258.881355240395 - 36869.76000015153) / 41258.881355240395 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Collections.ContainsFalse<Int32>.ImmutableSortedSet(Size: 512)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 36.49460145578721 < 39.655365546997814.
IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 2/9/2023 4:27:10 AM, 2/14/2023 2:39:38 AM falls between 2/4/2023 10:21:07 PM and 2/14/2023 2:39:38 AM.
IsImprovementStdDev: Marked as improvement because 40.72026905086423 (T) = (0 -37684.4274264379) / Math.Sqrt((673584.2143392202 / (299)) + (161763.53758866794 / (22))) is greater than 1.9674283869023508 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (22) - 2, .975) and 0.09576245422236819 = (41675.36241157716 - 37684.4274264379) / 41675.36241157716 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
_1.png)
_2.png)
_3.png)




_1.png)
_2.png)
_3.png)
_4.png)
_5.png)
_1.png)
_2.png)
_1.png)
_2.png)