Inductor: Increase multiplier to 3 for Inductor AMP benchmark correctness check (#109097)

leslie-fang-intel · pytorchmergebot · commit 55c19a3c6d38 · 2023-09-16T10:02:56.000Z
**Summary** As reported in #108333, we find some of the models have failed the benchmark's correctness check. However, the end-to-end model's accuracy ([test script](https://gist.github.com/leslie-fang-intel/aac8b3c2b450532fd0517c758bb845e0)) when comparing AMP with FP32 is within a difference of less than 0.1%. Thus, it's possible that the correctness check failures for these models are false alarms. We use multiplier of 3 instead of 2 in this PR to avoid these false alarms. Model end-to-end accuracy test results are: <html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta name=ProgId content=Excel.Sheet> <meta name=Generator content="Microsoft Excel 15"> <link id=Main-File rel=Main-File href="file:///C:/Users/jiahaofa/AppData/Local/Temp/msohtmlclip1/01/clip.htm"> <link rel=File-List href="file:///C:/Users/jiahaofa/AppData/Local/Temp/msohtmlclip1/01/clip_filelist.xml"> </head> <body link="#0563C1" vlink="#954F72"> SPR | | | | | | -- | -- | -- | -- | -- | -- | -- | FP32 Imperative TOP1 Accuracy | FP32 Imperative TOP5 Accuracy | BF16 AMP Inductor TOP1 Accuracy | BF16 AMP Inductor TOP5 Accuracy | BF16/FP32 Relative Loss TOP1 Accuracy | BF16/FP32 Relative Loss TOP5 Accuracy gluon_inception_v3 | 73.262 | 90.774 | 73.256 | 90.802 | -0.01% | 0.03% mobilenetv2_100 | 72.89 | 90.996 | 72.826 | 90.946 | -0.09% | -0.05% mobilenetv3_large_100 | 75.72 | 92.55 | 75.764 | 92.554 | 0.06% | 0.00% </body> </html> Pull Request resolved: #109097 Approved by: https://github.com/jgong5, https://github.com/jansel
diff --git a/torch/_dynamo/utils.py b/torch/_dynamo/utils.py
@@ -1062,7 +1062,13 @@ def to_tensor(t):
                     )
 
                 res_error = rmse(fp64_ref, res).item()
-                multiplier = 2.0
+
+                # In the case of using AMP (Automatic Mixed Precision), certain models have
+                # failed the benchmark's correctness check. However, the end-to-end model's
+                # accuracy when comparing AMP with FP32 is within a difference of less than 0.1%.
+                # Thus, it's possible that the correctness check failures for these models are
+                # false alarms. We use multiplier of 3 instead of 2 to avoid these false alarms.
+                multiplier = 3.0 if res.dtype == torch.bfloat16 else 2.0
 
                 if (
                     fp64_ref.numel() < 1000
