💊 CI failures summary and remediations

As of commit 5febc92 (more details on the Dr. CI page and at hud.pytorch.org/pr/61889):

• 1/2 failures introduced in this PR
• 1/2 broken upstream at merge base bf1c9aa from Jul 19 until Jul 20

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

pytorch_xla_linux_bionic_py3_6_clang9_test (1/1)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Jul 21 20:41:15   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_device_type.py", line 780, in test_wrapper
Jul 21 20:41:15     return test(*args, **kwargs)
Jul 21 20:41:15   File "/var/lib/jenkins/workspace/xla/test/test_ops.py", line 411, in test_reference_eager
Jul 21 20:41:15     self.compare_with_eager_reference(op, sample_input)
Jul 21 20:41:15   File "/var/lib/jenkins/workspace/xla/test/test_ops.py", line 402, in compare_with_eager_reference
Jul 21 20:41:15     self.assertEqual(actual, expected, exact_dtype=True, exact_device=False)
Jul 21 20:41:15   File "/var/lib/jenkins/workspace/xla/test/pytorch_test_base.py", line 608, in assertEqual
Jul 21 20:41:15     return DeviceTypeTestBase.assertEqual(self, x, y, *args, **kwargs)
Jul 21 20:41:15   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 1524, in assertEqual
Jul 21 20:41:15     super().assertTrue(result, msg=self._get_assert_msg(msg, debug_msg=debug_msg))
Jul 21 20:41:15 AssertionError: False is not true : Tensors failed to compare as equal!Attempted to compare equality of tensors with different sizes. Got sizes torch.Size([5, 5, 5]) and torch.Size([]).
Jul 21 20:41:15 
Jul 21 20:41:15 ----------------------------------------------------------------------
Jul 21 20:41:15 Ran 347 tests in 326.417s
Jul 21 20:41:15 
Jul 21 20:41:15 FAILED (failures=4)
Jul 21 20:41:15 
Jul 21 20:41:15 Generating XML reports...
Jul 21 20:41:16 + cleanup
Jul 21 20:41:16 + retcode=1
Jul 21 20:41:16 + set +x

XLA failure

Job pytorch_xla_linux_bionic_py3_6_clang9_test is failing. Please create an issue with title prefixed by [PT_BREAK] in pytorch/xla and link to to this PR. If you have questions, please reach out to @ailzhang / @dlibenzi / @JackCaoG.

🚧 1 fixed upstream failure:

These were probably caused by upstream breakages that were already fixed.

git fetch https://github.com/pytorch/pytorch viable/strict
git rebase FETCH_HEAD

• Linux CI (pytorch-linux-bionic-py3.8-gcc9-coverage) / render_test_results (default) from Jul 19 until Jul 20 (bf1c9aa - 018dc41)
  • 🔁 rerun

Preview docs built from this PR

Yes, will add a lint!

@ngimel has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@ngimel has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@ngimel has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@ngimel merged this pull request in 6284d2a.

@ngimel Any chance cudaStreamSynchronize could be completely removed from at::nonzero? We found it quite difficult to support for accelerators, including NVIDIA GPU.

Also cc @ezyang @ailzhang @asuhan @JackCaoG as we first identify this issue when supporting detection/segmentation models in PyTorch XLA but then find out it is mainly from the limitation (tensor shape must be concrete value) in PyTorch core.

Given the existing semantics of the operation, removing the synchronization is not possible.

What we should do, however, is support JAX's extension to the nonzero API https://jax.readthedocs.io/en/latest/_autosummary/jax.numpy.nonzero.html where an explicit size can be specified, giving an upper bound to the number of nonzero entries that will be returned (and zero padded if there aren't enough). Can you file an issue for this?

We would need this extension not only for nonzero, but also for indexing ops with mask, the most common situation when people encounter this particular sync is out = x[mask]

Given the existing semantics of the operation, removing the synchronization is not possible.

What we should do, however, is support JAX's extension to the nonzero API https://jax.readthedocs.io/en/latest/_autosummary/jax.numpy.nonzero.html where an explicit size can be specified, giving an upper bound to the number of nonzero entries that will be returned (and zero padded if there aren't enough). Can you file an issue for this?

Raised in #62320