🐛 Describe the bug

Originally discussed in transformers.

@tugsbayasgalan mentioned it might be a regression of torch 2.7.

cc @chauhang @penguinwu @voznesenskym @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @amjames @avikchaudhuri @gmagogsfm @zhxchen17 @tugsbayasgalan @angelayi @suo @ydwu4 @guangy10 and @anijain2305.

To Reproduce

1. Install torch 2.7 RC:

pip install torch==2.7.0 torchvision torchaudio --index-url https://download.pytorch.org/whl/test/cu126

2. Install transformers

git clone https://github.com/huggingface/transformers.git && cd transformers && git fetch origin && git checkout ci_with_torch_2.7_commit_0ef339ff1b63bb03a388c79bfbebec9085e10564 && pip install -e .[torch,testing]

3. Running test regarding torch.compile: get the error shown at the end, i.e. 7. (works if running with torch 2.6)

git checkout bcd1da9580ba1c6c4db019a91b1de9d88966e1fa && RUN_SLOW=1 python3 -m pytest -v tests/models/gemma3/test_modeling_gemma3.py::Gemma3Vision2TextModelTest::test_generate_compile_model_forward

4. Running test with a change in transformers: it works

git checkout 95eb065772215cab276678c45daf933599cfd337&& RUN_SLOW=1 python3 -m pytest -v tests/models/gemma3/test_modeling_gemma3.py::Gemma3Vision2TextModelTest::test_generate_compile_model_forward

The commit changes is here

5. another test with torch.export fails with torch 2.7 RC (it works if using strict=False in torch.export)

git checkout bcd1da9580ba1c6c4db019a91b1de9d88966e1fa && RUN_SLOW=1 python3 -m pytest -v tests/models/dpt/test_modeling_dpt_hybrid.py::DPTModelTest::test_torch_export

6. a change that would work

git checkout 49e7bd3e406e20beedec3c5d6d8be54aeb51daf5&& RUN_SLOW=1 python3 -m pytest -v tests/models/dpt/test_modeling_dpt_hybrid.py::DPTModelTest::test_torch_export

The commit changes is here

7. Error log from 3.

E               torch._dynamo.exc.Unsupported: Unexpected type in sourceless builder transformers.models.gemma3.configuration_gemma3.Gemma3TextConfig
E
E               from user code:
E                  File "/transformers/src/transformers/utils/generic.py", line 965, in wrapper
E                   output = func(self, *args, **kwargs)
E                 File "/transformers/src/transformers/utils/deprecation.py", line 172, in wrapped_func
E                   return func(*args, **kwargs)
E                 File "/transformers/src/transformers/models/gemma3/modeling_gemma3.py", line 1323, in forward
E                   causal_mask = self._update_causal_mask(
E                 File "/transformers/src/transformers/models/gemma3/modeling_gemma3.py", line 1117, in _update_causal_mask
E                   if self.config.text_config._attn_implementation == "flash_attention_2":
E                 File "/transformers/src/transformers/configuration_utils.py", line 210, in __getattribute__
E                   return super().__getattribute__(key)
E
E               Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"

Versions

--2025-04-10 07:41:28-- https://raw.githubusercontent.com/pytorch/pytorch/main/torch/utils/collect_env.py
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.111.133, 185.199.109.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 24440 (24K) [text/plain]
Saving to: 'collect_env.py'

collect_env.py 100%[====================================================================================================================================================================================>] 23.87K --.-KB/s in 0s

2025-04-10 07:41:28 (53.1 MB/s) - 'collect_env.py' saved [24440/24440]

root@d6af5b579bcb:/temp/transformers# python3 collect_env.py
Collecting environment information...
PyTorch version: 2.7.0+cu126
Is debug build: False
CUDA used to build PyTorch: 12.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.10.12 (main, Feb 4 2025, 14:57:36) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.10.234-225.921.amzn2.x86_64-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: Tesla T4
Nvidia driver version: 550.144.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
CPU family: 6
Model: 85
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
Stepping: 7
BogoMIPS: 4999.99
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 ss
e4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke avx512_vnni
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 128 KiB (4 instances)
L1i cache: 128 KiB (4 instances)
L2 cache: 4 MiB (4 instances)
L3 cache: 35.8 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-7
Vulnerability Gather data sampling: Unknown: Dependent on hypervisor status
Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported
Vulnerability L1tf: Mitigation; PTE Inversion
Vulnerability Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Vulnerable
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected

Versions of relevant libraries:
[pip3] intel-extension-for-pytorch==2.3.0
[pip3] mypy-extensions==1.0.0
[pip3] natten==0.17.4+torch250cu121
[pip3] numpy==1.24.3
[pip3] nvidia-cublas-cu12==12.6.4.1
[pip3] nvidia-cuda-cupti-cu12==12.6.80
[pip3] nvidia-cuda-nvrtc-cu12==12.6.77
[pip3] nvidia-cuda-runtime-cu12==12.6.77
[pip3] nvidia-cudnn-cu12==9.5.1.17
[pip3] nvidia-cufft-cu12==11.3.0.4
[pip3] nvidia-curand-cu12==10.3.7.77
[pip3] nvidia-cusolver-cu12==11.7.1.2
[pip3] nvidia-cusparse-cu12==12.5.4.2
[pip3] nvidia-cusparselt-cu12==0.6.3
[pip3] nvidia-nccl-cu12==2.26.2
[pip3] nvidia-nvjitlink-cu12==12.6.85
[pip3] nvidia-nvtx-cu12==12.6.77
[pip3] onnx==1.17.0
[pip3] onnxconverter-common==1.13.0
[pip3] onnxruntime==1.21.0
[pip3] onnxruntime-tools==1.7.0
[pip3] tf2onnx==1.16.1
[pip3] torch==2.7.0+cu126
[pip3] torchaudio==2.7.0+cu126
[pip3] torchvision==0.22.0+cu126
[pip3] triton==3.3.0
[conda] Could not collect