-
Notifications
You must be signed in to change notification settings - Fork 4
Closed
Description
Run Information
| Architecture | x64 |
|---|---|
| OS | Windows 10.0.19042 |
| Baseline | ac03fbd184b182a6632a50bbe70bc733e487264c |
| Compare | 4832236b4820ca83bbc9ce96003c2029b85c6ce2 |
| Diff | Diff |
Improvements in System.Tests.Perf_String
| Benchmark | Baseline | Test | Test/Base | Test Quality | Edge Detector | Baseline IR | Compare IR | IR Ratio | Baseline ETL | Compare ETL |
|---|---|---|---|---|---|---|---|---|---|---|
| ToUpperInvariant - Duration of single invocation | 48.78 ns | 27.70 ns | 0.57 | 0.01 | False | |||||
| Concat_str_str - Duration of single invocation | 29.97 ns | 27.73 ns | 0.93 | 0.05 | False | |||||
| ToLowerInvariant - Duration of single invocation | 48.28 ns | 27.65 ns | 0.57 | 0.03 | False | |||||
| ToLower - Duration of single invocation | 54.87 ns | 34.09 ns | 0.62 | 0.04 | False | |||||
| ToUpper - Duration of single invocation | 54.25 ns | 33.29 ns | 0.61 | 0.04 | False |
Repro
git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'Payloads
Histogram
System.Tests.Perf_String.ToUpperInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.")
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 27.70413843674101 < 45.865418126467624.
IsChangePoint: Marked as a change because one of 11/24/2022 8:21:24 PM, 11/29/2022 1:43:29 AM falls between 11/20/2022 12:28:28 PM and 11/29/2022 1:43:29 AM.
IsImprovementStdDev: Marked as improvement because 241.29521787786197 (T) = (0 -27.244834913532998) / Math.Sqrt((1.782055651107921 / (299)) + (0.061557660885357475 / (15))) is greater than 1.9675964973877862 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (15) - 2, .975) and 0.4704735836591698 = (51.45132343311997 - 27.244834913532998) / 51.45132343311997 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.Concat_str_str(size: 100)
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 27.72660231768547 < 28.508668179425687.
IsChangePoint: Marked as a change because one of 9/13/2022 12:23:36 PM, 9/23/2022 1:16:35 AM, 11/21/2022 9:59:44 PM, 11/29/2022 1:43:29 AM falls between 11/20/2022 12:28:28 PM and 11/29/2022 1:43:29 AM.
IsImprovementStdDev: Marked as improvement because 14.494934712396955 (T) = (0 -28.06160042430037) / Math.Sqrt((0.6494612449954009 / (299)) + (0.4854504050157415 / (33))) is greater than 1.9671786747533526 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (33) - 2, .975) and 0.06289463252159187 = (29.944978865940534 - 28.06160042430037) / 29.944978865940534 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.ToLowerInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.")
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 27.64643342074679 < 45.75298337236028.
IsChangePoint: Marked as a change because one of 11/11/2022 5:53:56 PM, 11/16/2022 6:51:59 PM, 11/24/2022 8:21:24 PM, 11/29/2022 1:43:29 AM falls between 11/20/2022 12:28:28 PM and 11/29/2022 1:43:29 AM.
IsImprovementStdDev: Marked as improvement because 153.2733072724317 (T) = (0 -27.24888821563144) / Math.Sqrt((6.463296630011602 / (299)) + (0.0708161262318981 / (15))) is greater than 1.9675964973877862 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (15) - 2, .975) and 0.4772228412073368 = (52.123333541507165 - 27.24888821563144) / 52.123333541507165 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.ToLower(s: "This is a much longer piece of text that might benefit more from vectorization.")
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 34.08580737527149 < 52.30128524165019.
IsChangePoint: Marked as a change because one of 11/11/2022 5:53:56 PM, 11/16/2022 11:33:39 PM, 11/24/2022 8:21:24 PM, 11/29/2022 1:43:29 AM falls between 11/20/2022 12:28:28 PM and 11/29/2022 1:43:29 AM.
IsImprovementStdDev: Marked as improvement because 94.94878090412887 (T) = (0 -33.99525752045803) / Math.Sqrt((17.900331925980527 / (299)) + (0.15693150926913657 / (15))) is greater than 1.9675964973877862 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (15) - 2, .975) and 0.4255169308525942 = (59.17538626666687 - 33.99525752045803) / 59.17538626666687 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
```#### System.Tests.Perf_String.ToUpper(s: "This is a much longer piece of text that might benefit more from vectorization.")
```log
Description of detection logic
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsRegressionBase: Marked as not a regression because the compare was not 5% greater than the baseline, or the value was too small.
IsImprovementBase: Marked as improvement because the compare was 5% less than the baseline, and the value was not too small.
IsImprovementCheck: Marked as improvement because the three check build points were 0.05 less than the baseline.
IsImprovementWindowed:Marked as improvement because 33.288464192246515 < 51.761065998791636.
IsChangePoint: Marked as a change because one of 11/11/2022 5:53:56 PM, 11/16/2022 11:33:39 PM, 11/24/2022 8:21:24 PM, 11/29/2022 1:43:29 AM falls between 11/20/2022 12:28:28 PM and 11/29/2022 1:43:29 AM.
IsImprovementStdDev: Marked as improvement because 91.62803508287547 (T) = (0 -33.96511062163668) / Math.Sqrt((20.14602569694683 / (299)) + (0.12731826921053743 / (15))) is greater than 1.9675964973877862 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (299) + (15) - 2, .975) and 0.4262933300524889 = (59.20292093648515 - 33.96511062163668) / 59.20292093648515 is greater than 0.05.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
Docs
Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository




