KEMBAR78
[Perf] Alpine/x64: 58 Improvements on 2/9/2023 12:40:40 PM · Issue #12943 · dotnet/perf-autofiling-issues · GitHub
Skip to content

[Perf] Alpine/x64: 58 Improvements on 2/9/2023 12:40:40 PM #12943

@performanceautofiler

Description

@performanceautofiler

Run Information

Architecture x64
OS alpine 3.15
Baseline 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4
Compare fe1607679b977b453d43c4de276968dd9ae6d5ec
Diff Diff

Improvements in System.Collections.CtorFromCollection<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableDictionary - Duration of single invocation 112.46 μs 105.35 μs 0.94 0.01 False
ConcurrentDictionary - Duration of single invocation 50.45 μs 21.37 μs 0.42 0.01 False
SortedDictionary - Duration of single invocation 51.44 μs 47.75 μs 0.93 0.01 False
ImmutableHashSet - Duration of single invocation 101.30 μs 92.31 μs 0.91 0.02 False

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollection&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CtorFromCollection<Int32>.ImmutableDictionary(Size: 512)


Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 105.35255485977947 < 105.94128253126411.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 7:27:15 AM falls between 2/5/2023 7:19:43 PM and 2/14/2023 7:27:15 AM.
IsImprovementStdDev: Marked as improvement because 23.049601540146526 (T) = (0 -102770.1392218111) / Math.Sqrt((2691054.2202400323 / (176)) + (2903124.8310453584 / (22))) is greater than 1.9721412216619414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (176) + (22) - 2, .975) and 0.07924446037690715 = (111615.01049874714 - 102770.1392218111) / 111615.01049874714 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorFromCollection&lt;Int32&gt;.ConcurrentDictionary(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 21.372497068533697 < 47.96721730939688.
IsChangePoint: Marked as a change because one of 2/9/2023 12:40:40 PM, 2/14/2023 7:27:15 AM falls between 2/5/2023 7:19:43 PM and 2/14/2023 7:27:15 AM.
IsImprovementStdDev: Marked as improvement because 483.1631132228493 (T) = (0 -21410.01874746862) / Math.Sqrt((241772.504528089 / (177)) + (49935.27820447392 / (22))) is greater than 1.9720790337782532 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (177) + (22) - 2, .975) and 0.5764025960629124 = (50543.31907720668 - 21410.01874746862) / 50543.31907720668 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorFromCollection&lt;Int32&gt;.SortedDictionary(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 47.746682404731 < 48.784586067184144.
IsChangePoint: Marked as a change because one of 11/29/2022 11:47:49 AM, 2/9/2023 4:27:10 AM, 2/14/2023 7:27:15 AM falls between 2/5/2023 7:19:43 PM and 2/14/2023 7:27:15 AM.
IsImprovementStdDev: Marked as improvement because 24.141666482139122 (T) = (0 -48054.06240096575) / Math.Sqrt((4911339.670790407 / (176)) + (48134.29390526623 / (23))) is greater than 1.9720790337782532 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (176) + (23) - 2, .975) and 0.08004782088544113 = (52235.391677877335 - 48054.06240096575) / 52235.391677877335 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CtorFromCollection&lt;Int32&gt;.ImmutableHashSet(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 92.30963643604915 < 96.30810780518843.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 7:27:15 AM falls between 2/5/2023 7:19:43 PM and 2/14/2023 7:27:15 AM.
IsImprovementStdDev: Marked as improvement because 29.218827221705517 (T) = (0 -91775.68990772369) / Math.Sqrt((2139438.1716021453 / (176)) + (2341554.70736618 / (23))) is greater than 1.9720790337782532 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (176) + (23) - 2, .975) and 0.09704687395480038 = (101639.48411108289 - 91775.68990772369) / 101639.48411108289 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS alpine 3.15
Baseline 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4
Compare fe1607679b977b453d43c4de276968dd9ae6d5ec
Diff Diff

Improvements in System.Globalization.Tests.StringSearch

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IsPrefix_FirstHalf - Duration of single invocation 130.65 ns 115.38 ns 0.88 0.01 False
IsSuffix_SecondHalf - Duration of single invocation 155.76 ns 130.98 ns 0.84 0.05 False
IsPrefix_FirstHalf - Duration of single invocation 130.58 ns 115.34 ns 0.88 0.01 False
IsPrefix_FirstHalf - Duration of single invocation 130.65 ns 115.41 ns 0.88 0.01 False
IsSuffix_SecondHalf - Duration of single invocation 155.72 ns 130.91 ns 0.84 0.03 False

graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.StringSearch*'

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, IgnoreNonSpace, False))


Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 115.38181117529587 < 124.06334117779082.
IsChangePoint: Marked as a change because one of 1/5/2023 9:29:47 PM, 1/18/2023 8:37:25 PM, 2/9/2023 4:27:10 AM, 2/14/2023 7:27:15 AM falls between 2/5/2023 7:19:43 PM and 2/14/2023 7:27:15 AM.
IsImprovementStdDev: Marked as improvement because 26.069166433606043 (T) = (0 -115.4424901116955) / Math.Sqrt((37.58226259418721 / (176)) + (0.005828155060240404 / (23))) is greater than 1.9720790337782532 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (176) + (23) - 2, .975) and 0.09454153524984525 = (127.49617415476901 - 115.4424901116955) / 127.49617415476901 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, IgnoreCase, False))

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 130.9839507948598 < 148.02393678828426.
IsChangePoint: Marked as a change because one of 12/1/2022 7:29:59 PM, 12/17/2022 10:44:50 PM, 1/5/2023 8:47:31 AM, 1/30/2023 11:52:26 AM, 2/9/2023 4:27:10 AM, 2/14/2023 7:27:15 AM falls between 2/5/2023 7:19:43 PM and 2/14/2023 7:27:15 AM.
IsImprovementStdDev: Marked as improvement because 17.37857359470476 (T) = (0 -131.64079832883016) / Math.Sqrt((137.3460828948471 / (299)) + (0.11259727424457044 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.08252619779660327 = (143.48180625177832 - 131.64079832883016) / 143.48180625177832 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, None, False))

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 115.33705423993571 < 124.06411666526401.
IsChangePoint: Marked as a change because one of 1/5/2023 9:29:47 PM, 1/18/2023 8:37:25 PM, 2/9/2023 4:27:10 AM, 2/14/2023 7:27:15 AM falls between 2/5/2023 7:19:43 PM and 2/14/2023 7:27:15 AM.
IsImprovementStdDev: Marked as improvement because 26.189534045291953 (T) = (0 -115.4297714482856) / Math.Sqrt((37.44526008945016 / (176)) + (0.0038139552224689508 / (23))) is greater than 1.9720790337782532 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (176) + (23) - 2, .975) and 0.0947718302582303 = (127.5145596509814 - 115.4297714482856) / 127.5145596509814 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (, None, False))

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 115.40575862636128 < 124.10852671919976.
IsChangePoint: Marked as a change because one of 1/5/2023 9:29:47 PM, 1/18/2023 8:37:25 PM, 2/9/2023 4:27:10 AM, 2/14/2023 7:27:15 AM falls between 2/5/2023 7:19:43 PM and 2/14/2023 7:27:15 AM.
IsImprovementStdDev: Marked as improvement because 26.0336849574237 (T) = (0 -115.4397346515852) / Math.Sqrt((37.57377342353674 / (176)) + (0.03457601257491241 / (23))) is greater than 1.9720790337782532 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (176) + (23) - 2, .975) and 0.09466697078028857 = (127.51079539324928 - 115.4397346515852) / 127.51079539324928 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (, IgnoreCase, False))

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 130.90982142857143 < 148.0047240456937.
IsChangePoint: Marked as a change because one of 11/29/2022 11:47:49 AM, 1/30/2023 11:52:26 AM, 2/9/2023 4:27:10 AM, 2/14/2023 7:27:15 AM falls between 2/5/2023 7:19:43 PM and 2/14/2023 7:27:15 AM.
IsImprovementStdDev: Marked as improvement because 12.804066081387074 (T) = (0 -131.63897266332842) / Math.Sqrt((134.9905495204924 / (176)) + (0.10234313040382935 / (23))) is greater than 1.9720790337782532 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (176) + (23) - 2, .975) and 0.07870688690607695 = (142.88500672848102 - 131.63897266332842) / 142.88500672848102 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS alpine 3.15
Baseline 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4
Compare fe1607679b977b453d43c4de276968dd9ae6d5ec
Diff Diff

Improvements in System.Collections.ContainsFalse<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableSortedSet - Duration of single invocation 28.42 μs 24.12 μs 0.85 0.01 False
SortedSet - Duration of single invocation 27.31 μs 23.67 μs 0.87 0.00 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsFalse&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsFalse<Int32>.ImmutableSortedSet(Size: 512)


Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 24.119118621794875 < 27.043979271687164.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 7:27:15 AM falls between 2/5/2023 7:19:43 PM and 2/14/2023 7:27:15 AM.
IsImprovementStdDev: Marked as improvement because 150.23338181539486 (T) = (0 -23669.60922981652) / Math.Sqrt((42861.80967792578 / (299)) + (20353.831751889342 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.1691130610945708 = (28487.160071378345 - 23669.60922981652) / 28487.160071378345 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsFalse&lt;Int32&gt;.SortedSet(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 23.67047927681268 < 25.888289303453377.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 7:27:15 AM falls between 2/5/2023 7:19:43 PM and 2/14/2023 7:27:15 AM.
IsImprovementStdDev: Marked as improvement because 138.2687016781119 (T) = (0 -23939.910188114336) / Math.Sqrt((46289.16333201491 / (176)) + (8678.622311545772 / (23))) is greater than 1.9720790337782532 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (176) + (23) - 2, .975) and 0.1275156928629827 = (27438.785995671493 - 23939.910188114336) / 27438.785995671493 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS alpine 3.15
Baseline 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4
Compare fe1607679b977b453d43c4de276968dd9ae6d5ec
Diff Diff

Improvements in ByteMark

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BenchStringSort - Duration of single invocation 1.15 secs 1.05 secs 0.91 0.00 False
BenchAssignJagged - Duration of single invocation 776.98 ms 733.18 ms 0.94 0.01 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'ByteMark*'

Payloads

Baseline
Compare

Histogram

ByteMark.BenchStringSort


Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.0505456563571427 < 1.0904002357460163.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 7:27:15 AM falls between 2/5/2023 7:19:43 PM and 2/14/2023 7:27:15 AM.
IsImprovementStdDev: Marked as improvement because 59.02071878061766 (T) = (0 -1053637435.0474602) / Math.Sqrt((158720197384879.47 / (299)) + (43928839982943.29 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.08047171842101832 = (1145845599.4830208 - 1053637435.0474602) / 1145845599.4830208 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### ByteMark.BenchAssignJagged

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 733.1787160666668 < 740.87619277.
IsChangePoint: Marked as a change because one of 2/9/2023 4:10:34 PM, 2/14/2023 7:27:15 AM falls between 2/5/2023 7:19:43 PM and 2/14/2023 7:27:15 AM.
IsImprovementStdDev: Marked as improvement because 31.657745798622308 (T) = (0 -732962557.0306461) / Math.Sqrt((106355059489392.9 / (178)) + (28430009807236.355 / (21))) is greater than 1.9720790337782532 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (178) + (21) - 2, .975) and 0.05690084612783028 = (777185043.6099467 - 732962557.0306461) / 777185043.6099467 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS alpine 3.15
Baseline 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4
Compare fe1607679b977b453d43c4de276968dd9ae6d5ec
Diff Diff

Improvements in System.Tests.Perf_Double

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Parse - Duration of single invocation 106.03 ns 86.42 ns 0.82 0.02 False
TryParse - Duration of single invocation 105.05 ns 99.42 ns 0.95 0.02 False
TryParse - Duration of single invocation 101.26 ns 94.66 ns 0.93 0.03 False
Parse - Duration of single invocation 107.78 ns 88.63 ns 0.82 0.01 False

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Double*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Double.Parse(value: "1.7976931348623157e+308")


Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 86.415774728392 < 100.66060473063416.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 7:27:15 AM falls between 2/5/2023 7:19:43 PM and 2/14/2023 7:27:15 AM.
IsImprovementStdDev: Marked as improvement because 25.185048579495312 (T) = (0 -92.1525876252544) / Math.Sqrt((1.8287100435895554 / (184)) + (5.617528475182294 / (23))) is greater than 1.9716034992643237 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (184) + (23) - 2, .975) and 0.12110010331213447 = (104.849924288911 - 92.1525876252544) / 104.849924288911 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Double.TryParse(value: "-1.7976931348623157e+308")

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 99.42237991518 < 99.76880559625542.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 7:27:15 AM falls between 2/5/2023 7:19:43 PM and 2/14/2023 7:27:15 AM.
IsImprovementStdDev: Marked as improvement because 27.088340860480123 (T) = (0 -91.34426223663898) / Math.Sqrt((2.6105219491728575 / (278)) + (6.650202025013274 / (23))) is greater than 1.9679296690655894 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (278) + (23) - 2, .975) and 0.1394368511430958 = (106.14475225667356 - 91.34426223663898) / 106.14475225667356 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Double.TryParse(value: "1.7976931348623157e+308")

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 94.65690724420142 < 96.45732403271971.
IsChangePoint: Marked as a change because one of 1/13/2023 4:37:08 PM, 2/9/2023 4:27:10 AM, 2/14/2023 7:27:15 AM falls between 2/5/2023 7:19:43 PM and 2/14/2023 7:27:15 AM.
IsImprovementStdDev: Marked as improvement because 28.350379122285045 (T) = (0 -89.5921318060243) / Math.Sqrt((3.244979833550474 / (299)) + (5.374637199448164 / (23))) is greater than 1.9674049737480528 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (23) - 2, .975) and 0.13530675994757752 = (103.61146318270364 - 89.5921318060243) / 103.61146318270364 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Double.Parse(value: "-1.7976931348623157e+308")

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 88.62537096593823 < 102.39567916584159.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 7:27:15 AM falls between 2/5/2023 7:19:43 PM and 2/14/2023 7:27:15 AM.
IsImprovementStdDev: Marked as improvement because 26.024538683934907 (T) = (0 -94.48285144214587) / Math.Sqrt((1.2180688987601382 / (200)) + (5.264791393410541 / (23))) is greater than 1.9707562704893595 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (200) + (23) - 2, .975) and 0.11779533200071511 = (107.09856212438729 - 94.48285144214587) / 107.09856212438729 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS alpine 3.15
Baseline 765e3dbb2d3aa2b114d8c07e44ae2ca926df16b4
Compare fe1607679b977b453d43c4de276968dd9ae6d5ec
Diff Diff

Improvements in System.Collections.TryGetValueFalse<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SortedDictionary - Duration of single invocation 39.89 μs 35.79 μs 0.90 0.02 False
ImmutableSortedDictionary - Duration of single invocation 27.00 μs 24.00 μs 0.89 0.02 False
ConcurrentDictionary - Duration of single invocation 2.26 μs 1.59 μs 0.70 0.12 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryGetValueFalse&lt;Int32, Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.TryGetValueFalse<Int32, Int32>.SortedDictionary(Size: 512)


Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 35.78999876952489 < 37.792382288119015.
IsChangePoint: Marked as a change because one of 11/29/2022 11:47:49 AM, 2/9/2023 4:27:10 AM, 2/14/2023 7:27:15 AM falls between 2/5/2023 7:19:43 PM and 2/14/2023 7:27:15 AM.
IsImprovementStdDev: Marked as improvement because 25.769584041891722 (T) = (0 -34990.88178213833) / Math.Sqrt((7980060.287465244 / (176)) + (107521.1874714204 / (23))) is greater than 1.9720790337782532 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (176) + (23) - 2, .975) and 0.1414136556603103 = (40754.062783340276 - 34990.88178213833) / 40754.062783340276 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.TryGetValueFalse&lt;Int32, Int32&gt;.ImmutableSortedDictionary(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 23.996671505376344 < 25.66422042298272.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/14/2023 7:27:15 AM falls between 2/5/2023 7:19:43 PM and 2/14/2023 7:27:15 AM.
IsImprovementStdDev: Marked as improvement because 40.86584341027516 (T) = (0 -23041.352222932466) / Math.Sqrt((193039.6375189821 / (176)) + (211028.39886229963 / (23))) is greater than 1.9720790337782532 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (176) + (23) - 2, .975) and 0.15236586564212892 = (27183.1339595444 - 23041.352222932466) / 27183.1339595444 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.TryGetValueFalse&lt;Int32, Int32&gt;.ConcurrentDictionary(Size: 512)

```log

Description of detection logic

IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 1.5858755983225379 < 2.1346860723141075.
IsChangePoint: Marked as a change because one of 2/9/2023 12:40:40 PM, 2/14/2023 7:27:15 AM falls between 2/5/2023 7:19:43 PM and 2/14/2023 7:27:15 AM.
IsImprovementStdDev: Marked as improvement because 44.87680973636697 (T) = (0 -1600.6065238941605) / Math.Sqrt((9993.937580192412 / (176)) + (3525.7777294950224 / (22))) is greater than 1.9721412216619414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (176) + (22) - 2, .975) and 0.2923162077775273 = (2261.7538249215436 - 1600.6065238941605) / 2261.7538249215436 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions