Fix mismatched tensor metadata between FakeTensor and Intel XPU concrete tensor when running `F.logsigmoid` #141333

chunhuanMeng · 2024-11-22T07:28:59Z

Fixes #141332
F.logsigmoid will return two outputs: output and buffer.
For F.logsigmoid cpu path, it will use buffer to store some intermediate values and use them when computing gradients, so it returns a buffer tensor with nonzero size. For cuda and xpu paths, buffer is useless, so the buffer tensor size of xpu F.logsigmoid will be zero, just like cuda. The root cause of the issue is that the codes in decompositions.py (ref:https://github.com/pytorch/pytorch/blob/main/torch/_decomp/decompositions.py#L2803) only handle the cuda cases, when the a fake tensor with device is xpu run to here, it will use the cpu path and return a buffer with nonzero size, which is conflict to the implementation of intel xpu concrete tensor. Therefore this pr add conditions to handle xpu cases. Make sure the two returned buffer sizes match each other.

cc @gujinghui @EikanWang @fengyuan14 @guangyey

pytorch-bot · 2024-11-22T07:29:02Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/141333

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures, 5 Unrelated Failures

As of commit d47f443 with merge base 5deca07 ():

NEW FAILURES - The following jobs have failed:

xpu / linux-jammy-xpu-2025.0-py3.9 / test (default, 1, 4, linux.idc.xpu) (gh)
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
xpu / linux-jammy-xpu-2025.0-py3.9 / test (default, 2, 4, linux.idc.xpu) (gh)
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
xpu / linux-jammy-xpu-2025.0-py3.9 / test (default, 4, 4, linux.idc.xpu) (gh)
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

inductor-rocm / rocm6.2-py3.10-inductor / test (inductor, 1, 2, linux.rocm.gpu.2) (gh) (similar failure)
##[error]Credentials could not be loaded, please check your action inputs: Could not load credentials from any providers
inductor-rocm / rocm6.2-py3.10-inductor / test (inductor, 2, 2, linux.rocm.gpu.2) (gh) (similar failure)
##[error]Credentials could not be loaded, please check your action inputs: Could not load credentials from any providers
xpu / linux-jammy-xpu-2025.0-py3.9 / test (default, 3, 4, linux.idc.xpu) (gh) (matched linux rule in flaky-rules.json)
Error response from daemon: page not found

UNSTABLE - The following jobs failed but were likely due to flakiness present on trunk and has been marked as unstable:

inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu, unstable) (gh) (#141703)
convnext_base
trunk / win-vs2019-cpu-py3 / test (default, 1, 3, lf.windows.4xlarge.nonephemeral, unstable) (gh)
test_utils.py::TestDeviceUtilsCPU::test_device_mode_ops_sparse_mm_reduce_cpu_float16

This comment was automatically generated by Dr. CI and updates every 15 minutes.

chunhuanMeng · 2024-11-22T07:32:55Z

@pytorchbot label "topic: not user facing"

chunhuanMeng · 2024-11-22T07:34:03Z

@pytorchbot label "module: xpu"

guangyey · 2024-11-22T07:40:11Z

Could you add a UT for XPU?

chunhuanMeng · 2024-11-22T07:45:52Z

Could you add a UT for XPU?

Will test these cases in project torch-xpu-ops

EikanWang · 2024-11-26T07:16:25Z

@pytorchbot rebase

pytorchmergebot · 2024-11-26T07:17:58Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-11-26T07:18:01Z

Successfully rebased main onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout main && git pull --rebase)

EikanWang

Pls. fix the lint issue.

guangyey · 2024-11-27T04:00:04Z

torch/_decomp/decompositions_for_jvp.py

    min = torch.minimum(self.new_zeros(()), self)
    z = torch.exp(-torch.abs(self))
-    if self.is_cuda:
+    if self.is_cuda or self.is_xpu:


Add is_xpu to tensor doc, refer to

pytorch/torch/_tensor_docs.py

Lines 6688 to 6693 in 6e61ff4

add_docstr_all(

"is_cuda",

r"""

Is ``True`` if the Tensor is stored on the GPU, ``False`` otherwise.

""",

)

There are is_xpu in this file. ref:https://github.com/pytorch/pytorch/blob/main/torch/_tensor_docs.py#L6716

Add is_xpu to pyi, refer to

pytorch/tools/pyi/gen_pyi.py

Lines 1205 to 1208 in 5ca75ac

"is_cpu": ["is_cpu: _bool"],

"is_cuda": ["is_cuda: _bool"],

"is_leaf": ["is_leaf: _bool"],

"is_nested": ["is_nested: _bool"],

guangyey · 2024-11-28T02:56:37Z

@pytorchbot rebase

pytorchmergebot · 2024-11-28T02:58:14Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-11-28T02:58:17Z

Successfully rebased main onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout main && git pull --rebase)

EikanWang · 2024-11-29T07:40:20Z

@pytorchbot rebase

pytorchmergebot · 2024-11-29T07:41:48Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-11-29T07:41:51Z

Successfully rebased main onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout main && git pull --rebase)

EikanWang · 2024-12-02T01:44:16Z

@pytorchbot rebase

pytorchmergebot · 2024-12-02T01:45:51Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-12-02T01:45:54Z

Successfully rebased main onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout main && git pull --rebase)

guangyey · 2024-12-02T07:33:45Z

@pytorchbot rebase

pytorchmergebot · 2024-12-02T07:35:21Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-12-02T07:35:24Z

Tried to rebase and push PR #141333, but it was already up to date. Try rebasing against main by issuing:
@pytorchbot rebase -b main

ezyang · 2024-12-02T22:07:24Z

@pytorchbot merge -f "this looks fine to force land"

pytorchmergebot · 2024-12-02T22:09:02Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…ete tensor when running `F.logsigmoid` (pytorch#141333) Fixes pytorch#141332 `F.logsigmoid` will return two outputs: `output` and `buffer`. For `F.logsigmoid` cpu path, it will use buffer to store some intermediate values and use them when computing gradients, so it returns a `buffer` tensor with nonzero size. For cuda and xpu paths, buffer is useless, so the `buffer ` tensor size of xpu `F.logsigmoid` will be zero, just like cuda. The root cause of the issue is that the codes in `decompositions.py` (ref:https://github.com/pytorch/pytorch/blob/main/torch/_decomp/decompositions.py#L2803) only handle the cuda cases, when the a fake tensor with device is xpu run to here, it will use the cpu path and return a `buffer` with nonzero size, which is conflict to the implementation of intel xpu concrete tensor. Therefore this pr add conditions to handle xpu cases. Make sure the two returned buffer sizes match each other. Pull Request resolved: pytorch#141333 Approved by: https://github.com/guangyey, https://github.com/EikanWang, https://github.com/ezyang

pytorchbot added the open source label Nov 22, 2024

pytorch-bot bot added the topic: not user facing topic category label Nov 22, 2024

pytorch-bot bot added the module: xpu Intel XPU related issues label Nov 22, 2024

guangyey requested a review from EikanWang November 22, 2024 07:35

guangyey added the ciflow/xpu Run XPU CI tasks label Nov 22, 2024

guangyey approved these changes Nov 22, 2024

View reviewed changes

chunhuanMeng mentioned this pull request Nov 22, 2024

shapes are not equal with FakeTensor intel/torch-xpu-ops#1078

Closed

EikanWang approved these changes Nov 26, 2024

View reviewed changes

pytorchmergebot force-pushed the main branch from 112253c to 9469010 Compare November 26, 2024 07:18

pytorch-bot bot added the ciflow/inductor label Nov 26, 2024

chunhuanMeng changed the title ~~Add conditions to allow XPU to use zero buffer~~ Fix mismatched tensor metadata between FakeTensor and Intel XPU concrete tensors when running F.logsigmoid Nov 26, 2024

chunhuanMeng changed the title ~~Fix mismatched tensor metadata between FakeTensor and Intel XPU concrete tensors when running F.logsigmoid~~ Fix mismatched tensor metadata between FakeTensor and Intel XPU concrete tensor when running F.logsigmoid Nov 26, 2024

EikanWang requested changes Nov 26, 2024

View reviewed changes

colesbury added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Nov 26, 2024

guangyey reviewed Nov 27, 2024

View reviewed changes

pytorchmergebot force-pushed the main branch from 63a480e to 92fa4de Compare November 28, 2024 02:58

guangyey added the ciflow/trunk Trigger trunk jobs on your pull request label Nov 28, 2024

guangyey added this to the 2.6.0 milestone Nov 28, 2024

EikanWang approved these changes Nov 29, 2024

View reviewed changes

pytorchmergebot force-pushed the main branch from 92fa4de to 7ce274e Compare November 29, 2024 07:41

guangyey requested review from atalman, ezyang and malfet November 29, 2024 10:28

chunhuanMeng added 3 commits December 2, 2024 01:45

allow xpu

3ca0f6e

allow xpu

29fd845

add is_xpu interface

d47f443

pytorchmergebot force-pushed the main branch from 7ce274e to d47f443 Compare December 2, 2024 01:45

ezyang approved these changes Dec 2, 2024

View reviewed changes

pytorchmergebot added the merging label Dec 2, 2024

pytorchmergebot added the Merged label Dec 2, 2024

pytorchmergebot closed this in 1f3d889 Dec 2, 2024

pytorchmergebot removed the merging label Dec 2, 2024

atalman mentioned this pull request Jan 13, 2025

Release 2.6.0 validations checklist and cherry-picks #144503

Closed

73 tasks

	add_docstr_all(
	"is_cuda",
	r"""
	Is ``True`` if the Tensor is stored on the GPU, ``False`` otherwise.
	""",
	)

	"is_cpu": ["is_cpu: _bool"],
	"is_cuda": ["is_cuda: _bool"],
	"is_leaf": ["is_leaf: _bool"],
	"is_nested": ["is_nested: _bool"],

Fix mismatched tensor metadata between FakeTensor and Intel XPU concrete tensor when running F.logsigmoid #141333

Fix mismatched tensor metadata between FakeTensor and Intel XPU concrete tensor when running F.logsigmoid #141333

Uh oh!

Conversation

chunhuanMeng commented Nov 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Nov 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/141333

❌ 3 New Failures, 5 Unrelated Failures

Uh oh!

chunhuanMeng commented Nov 22, 2024

Uh oh!

chunhuanMeng commented Nov 22, 2024

Uh oh!

guangyey commented Nov 22, 2024

Uh oh!

chunhuanMeng commented Nov 22, 2024

Uh oh!

EikanWang commented Nov 26, 2024

Uh oh!

pytorchmergebot commented Nov 26, 2024

Uh oh!

pytorchmergebot commented Nov 26, 2024

Uh oh!

EikanWang left a comment

Choose a reason for hiding this comment

Uh oh!

guangyey Nov 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chunhuanMeng Nov 28, 2024

Choose a reason for hiding this comment

Uh oh!

guangyey Nov 28, 2024

Choose a reason for hiding this comment

Uh oh!

guangyey commented Nov 28, 2024

Uh oh!

pytorchmergebot commented Nov 28, 2024

Uh oh!

pytorchmergebot commented Nov 28, 2024

Uh oh!

EikanWang commented Nov 29, 2024

Uh oh!

pytorchmergebot commented Nov 29, 2024

Uh oh!

pytorchmergebot commented Nov 29, 2024

Uh oh!

EikanWang commented Dec 2, 2024

Uh oh!

pytorchmergebot commented Dec 2, 2024

Uh oh!

pytorchmergebot commented Dec 2, 2024

Uh oh!

guangyey commented Dec 2, 2024

Uh oh!

pytorchmergebot commented Dec 2, 2024

Uh oh!

pytorchmergebot commented Dec 2, 2024

Uh oh!

ezyang commented Dec 2, 2024

Uh oh!

pytorchmergebot commented Dec 2, 2024

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Fix mismatched tensor metadata between FakeTensor and Intel XPU concrete tensor when running `F.logsigmoid` #141333

Fix mismatched tensor metadata between FakeTensor and Intel XPU concrete tensor when running `F.logsigmoid` #141333

chunhuanMeng commented Nov 22, 2024 •

edited

Loading

pytorch-bot bot commented Nov 22, 2024 •

edited

Loading

guangyey Nov 27, 2024 •

edited

Loading