-
Notifications
You must be signed in to change notification settings - Fork 4
Closed
Description
Run Information
| Architecture | x64 |
|---|---|
| OS | alpine 3.15 |
| Baseline | ac03fbd184b182a6632a50bbe70bc733e487264c |
| Compare | 4832236b4820ca83bbc9ce96003c2029b85c6ce2 |
| Diff | Diff |
Improvements in System.Tests.Perf_String
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| ToLowerInvariant - Duration of single invocation | 44.41 ns | 23.78 ns | 0.54 | 0.07 | False | |||||
| ToUpper - Duration of single invocation | 47.54 ns | 30.28 ns | 0.64 | 0.16 | False | |||||
| ToLower - Duration of single invocation | 47.92 ns | 30.10 ns | 0.63 | 0.12 | False | |||||
| CtorCharCount - Duration of single invocation | 10.81 ns | 9.62 ns | 0.89 | 0.09 | False | |||||
| ToUpperInvariant - Duration of single invocation | 40.31 ns | 23.94 ns | 0.59 | 0.06 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'Payloads
Histogram
System.Tests.Perf_String.ToLowerInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.")
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 23.779461584760273 < 38.097726695373446.
IsChangePoint: Marked as a change because one of 11/24/2022 8:21:24 PM, 11/29/2022 1:43:29 AM falls between 11/20/2022 12:28:28 PM and 11/29/2022 1:43:29 AM.
IsImprovementStdDev: Marked as improvement because 176.19668218180027 (T) = (0 -23.88412690355911) / Math.Sqrt((1.439354111635046 / (299)) + (0.0800362272167984 / (15))) is greater than 1.9675964973877862 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (15) - 2, .975) and 0.4263468718045812 = (41.63513755898638 - 23.88412690355911) / 41.63513755898638 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.ToUpper(s: "This is a much longer piece of text that might benefit more from vectorization.")
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 30.281007624164626 < 45.82022974836062.
IsChangePoint: Marked as a change because one of 11/24/2022 8:21:24 PM, 11/29/2022 1:43:29 AM falls between 11/20/2022 12:28:28 PM and 11/29/2022 1:43:29 AM.
IsImprovementStdDev: Marked as improvement because 77.56705377190865 (T) = (0 -31.031081314072047) / Math.Sqrt((2.124862787030778 / (299)) + (0.713584971783942 / (15))) is greater than 1.9675964973877862 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (15) - 2, .975) and 0.3688891607324463 = (49.168988049841914 - 31.031081314072047) / 49.168988049841914 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.ToLower(s: "This is a much longer piece of text that might benefit more from vectorization.")
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 30.104827840951614 < 45.89687030275319.
IsChangePoint: Marked as a change because one of 11/24/2022 8:21:24 PM, 11/29/2022 1:43:29 AM falls between 11/20/2022 12:28:28 PM and 11/29/2022 1:43:29 AM.
IsImprovementStdDev: Marked as improvement because 30.53962866252476 (T) = (0 -31.740327595964168) / Math.Sqrt((1.9168907671293915 / (299)) + (4.796018315261271 / (15))) is greater than 1.9675964973877862 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (15) - 2, .975) and 0.3546254595072389 = (49.181251512849514 - 31.740327595964168) / 49.181251512849514 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.CtorCharCount(size: 10)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 9.62352235842726 < 10.319935938927934.
IsChangePoint: Marked as a change because one of 11/23/2022 8:02:34 PM, 11/29/2022 1:43:29 AM falls between 11/20/2022 12:28:28 PM and 11/29/2022 1:43:29 AM.
IsImprovementStdDev: Marked as improvement because 12.287370409546117 (T) = (0 -9.749865599623584) / Math.Sqrt((0.08869693275739093 / (299)) + (0.08627999317966933 / (21))) is greater than 1.967451947860886 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (21) - 2, .975) and 0.07718921182392045 = (10.565400539902697 - 9.749865599623584) / 10.565400539902697 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.ToUpperInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.")
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 23.941916612861366 < 37.95455504560814.
IsChangePoint: Marked as a change because one of 11/24/2022 8:21:24 PM, 11/29/2022 1:43:29 AM falls between 11/20/2022 12:28:28 PM and 11/29/2022 1:43:29 AM.
IsImprovementStdDev: Marked as improvement because 65.4140410880732 (T) = (0 -24.436214223541604) / Math.Sqrt((1.4712127062823668 / (299)) + (0.9286786357331351 / (15))) is greater than 1.9675964973877862 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (15) - 2, .975) and 0.408996996717935 = (41.347022075755945 - 24.436214223541604) / 41.347022075755945 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository




