MemcpyFromHost and MemcpyToHost support for plugin EPs #26088

edgchen1 · 2025-09-18T23:00:12Z

Description

Add support for MemcpyFromHost and MemcpyToHost ops with plugin EPs.

Add CPU EP fallback kernels for the memcpy ops. These are generic implementations using a data transfer manager.
Update SessionState::PopulateKernelCreateInfo() to fall back to CPU memcpy kernels if a node's assigned provider doesn't have them.
Update MemcpyTransformer to determine whether providers are CPU-based or compatible with other providers by looking at the device type instead of matching against a hardcoded list of provider types. This accommodates plugin EPs, where the provider type can't be hardcoded.

Motivation and Context

Allow plugin EPs to work with models where memcpy ops are required (i.e., models where connected nodes are not fully assigned to the plugin EP).

…for partially supported model

github-actions

You can commit the suggested changes from lintrunner.

onnxruntime/core/optimizer/transformer_memcpy.cc

onnxruntime/test/testdata/add_mul_add.py

onnxruntime/core/optimizer/transformer_memcpy.h

…provider.h

onnxruntime/test/autoep/library/ep.cc

onnxruntime/core/framework/utils.h

onnxruntime/test/autoep/library/ep_stream_support.cc

onnxruntime/core/optimizer/transformer_memcpy.cc

onnxruntime/core/providers/cpu/cpu_execution_provider.cc

onnxruntime/test/autoep/library/ep_stream_support.cc

…1/memcpy_op_support_for_plugin_eps_fix

…pport_for_plugin_eps_fix

skottmckay

Add support for `MemcpyFromHost` and `MemcpyToHost` ops with plugin EPs. - Add CPU EP fallback kernels for the memcpy ops. These are generic implementations using a data transfer manager. - Update `SessionState::PopulateKernelCreateInfo()` to fall back to CPU memcpy kernels if a node's assigned provider doesn't have them. - Update `MemcpyTransformer` to determine whether providers are CPU-based or compatible with other providers by looking at the device type instead of matching against a hardcoded list of provider types. This accommodates plugin EPs, where the provider type can't be hardcoded.  Allow plugin EPs to work with models where memcpy ops are required (i.e., models where connected nodes are not fully assigned to the plugin EP).

### Description Cherry-pick the following PRs into the ORT 1.23.1 branch: - Fix Attention GQA implementation on CPU - **MANUAL MERGE**: see #26057 - main merge date: Sept 15, 11:33am - pr: #25966 - commit: d530b29 - Address edge GetMemInfo edge cases - main merge date: Sept 16, 10:32am - pr: #26021 - commit: d251f3a - Implement new Python APIs - main merge date: Sept 17, 11:44am - pr: #25999 - commit: abc63e8 - MemcpyFromHost and MemcpyToHost support for plugin EPs - **MERGE CONFLICT** on file onnxruntime/test/optimizer/transpose_optimizer_test.cc. Conflicts with #25689 - main merge date: Sept 23, 10:42am - pr: #26088 - commit: 4545732 - [TRT RTX EP] Fix bug for generating the correct subgraph in GetCapability #26132 - main merge date: Sept 23, 8:54pm - pr: #26132 - commit: 72e56e7 ### Motivation and Context  --------- Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com> Co-authored-by: Chi Lo <54722500+chilo-ms@users.noreply.github.com>

snnn · 2025-09-25T22:45:16Z

This PR has been cherry-picked into the rel-1.23.1 branch in PR #26140. Removing the release:1.23.1 label.

* ORT 1.23.1 cherrypick 1 [REDO] (microsoft#26140) ### Description Cherry-pick the following PRs into the ORT 1.23.1 branch: - Fix Attention GQA implementation on CPU - **MANUAL MERGE**: see microsoft#26057 - main merge date: Sept 15, 11:33am - pr: microsoft#25966 - commit: d530b29 - Address edge GetMemInfo edge cases - main merge date: Sept 16, 10:32am - pr: microsoft#26021 - commit: d251f3a - Implement new Python APIs - main merge date: Sept 17, 11:44am - pr: microsoft#25999 - commit: abc63e8 - MemcpyFromHost and MemcpyToHost support for plugin EPs - **MERGE CONFLICT** on file onnxruntime/test/optimizer/transpose_optimizer_test.cc. Conflicts with microsoft#25689 - main merge date: Sept 23, 10:42am - pr: microsoft#26088 - commit: 4545732 - [TRT RTX EP] Fix bug for generating the correct subgraph in GetCapability microsoft#26132 - main merge date: Sept 23, 8:54pm - pr: microsoft#26132 - commit: 72e56e7 ### Motivation and Context  --------- Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com> Co-authored-by: Chi Lo <54722500+chilo-ms@users.noreply.github.com> * ORT 1.23.1 cherrypick 2 (microsoft#26182) ### Description Adds the following commits to the `rel-1.23.1` branch for ORT 1.23.1: - add session_id_ to LogEvaluationStart/Stop, LogSessionCreationStart - main merge date: July 31, 1:05am - pr: microsoft#25590 - commit: e753643 - [build] fix WebAssembly build on macOS/arm64 - main merge date: Aug 5, 8:07am - pr: microsoft#25653 - commit: 53f152b - [CPU] MoE Kernel (microsoft#25958) - main merge date: Sept 10, 4:54pm - pr: microsoft#25958 - commit: 930e640 - [CPU] Block-wise QMoE kernel for CPU - main merge date: Sept 15, 8:32am - pr: microsoft#26009 - commit: 5d17734 - [C#] Implement missing APIs - main merge date: Sept 24, 10:50am - pr: microsoft#26101 - commit: 35dcab5 - Regenerate test model with ONNX IR < 12 - main merge date: Sept 24, 2:50pm - pr: microsoft#26149 - commit: 88f2652 - [CPU] Fix compilation errors because of unused variables - main merge date: Sept 25, 1:21pm - pr: microsoft#26147 - commit: 42fcd71 - [EP ABI] Check if nodes specified in GetCapability() have already been assigned - main merge date: Sept 26, 1:24am - pr: microsoft#26156 - commit: 67d3ba0 - [QNN EP] Add dynamic option to set HTP performance mode - main merge date: Sept 26, 11:55am - pr: microsoft#26135 - commit: 6cc40fd --------- Co-authored-by: xieofxie <xieofxie@126.com> Co-authored-by: hualxie <hualxie@microsoft.com> Co-authored-by: Yulong Wang <7679871+fs-eire@users.noreply.github.com> Co-authored-by: Akshay Sonawane <111780983+apsonawane@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com> Co-authored-by: quic-tirupath <quic_tirupath@quicinc.com> Co-authored-by: quic-ashwshan <quic_ashwshan@quicinc.com> --------- Co-authored-by: Adrian Lizarraga <adlizarraga@microsoft.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com> Co-authored-by: Chi Lo <54722500+chilo-ms@users.noreply.github.com> Co-authored-by: xieofxie <xieofxie@126.com> Co-authored-by: hualxie <hualxie@microsoft.com> Co-authored-by: Yulong Wang <7679871+fs-eire@users.noreply.github.com> Co-authored-by: Akshay Sonawane <111780983+apsonawane@users.noreply.github.com> Co-authored-by: quic-tirupath <quic_tirupath@quicinc.com> Co-authored-by: quic-ashwshan <quic_ashwshan@quicinc.com>

edgchen1 added 2 commits September 18, 2025 14:59

save work - add memcpy kernel fallback in CPU EP, add plugin EP test …

7ec9e71

…for partially supported model

move ProviderIsCpuBased back to utils, add TODO

a668ac1

github-actions bot reviewed Sep 18, 2025

View reviewed changes

onnxruntime/core/optimizer/transformer_memcpy.cc Outdated Show resolved Hide resolved

github-advanced-security bot found potential problems Sep 18, 2025

View reviewed changes

onnxruntime/test/testdata/add_mul_add.py Fixed Show fixed Hide fixed

formatting

df0fcce

adrianlizarraga reviewed Sep 18, 2025

View reviewed changes

onnxruntime/core/optimizer/transformer_memcpy.h Show resolved Hide resolved

edgchen1 added 3 commits September 18, 2025 17:02

fix tests using old MemcpyTransformer ctor

9bf9e6b

move kInternalTestingExecutionProvider to internal_testing_execution_…

0e6cda0

…provider.h

add TODO for updating compatibility checks to use provider device

b42bcef

edgchen1 commented Sep 19, 2025

View reviewed changes

onnxruntime/test/autoep/library/ep.cc Show resolved Hide resolved

edgchen1 commented Sep 19, 2025

View reviewed changes

onnxruntime/core/framework/utils.h Show resolved Hide resolved

edgchen1 marked this pull request as ready for review September 19, 2025 00:58

edgchen1 requested review from adrianlizarraga and skottmckay September 19, 2025 00:58

edgchen1 changed the title ~~[WIP] MemcpyFromHost and MemcpyToHost support for plugin EPs~~ MemcpyFromHost and MemcpyToHost support for plugin EPs Sep 19, 2025

put helper fn in ifdef

32fb2a0

edgchen1 added the release:1.23.1 label Sep 19, 2025

edgchen1 commented Sep 19, 2025

View reviewed changes

onnxruntime/test/autoep/library/ep_stream_support.cc Show resolved Hide resolved

skottmckay reviewed Sep 19, 2025

View reviewed changes

onnxruntime/core/optimizer/transformer_memcpy.cc Outdated Show resolved Hide resolved

onnxruntime/core/providers/cpu/cpu_execution_provider.cc Show resolved Hide resolved

onnxruntime/test/autoep/library/ep_stream_support.cc Show resolved Hide resolved

edgchen1 added 6 commits September 19, 2025 11:47

also handle sequence types

ae2ce5b

move memcpy kernel fallback to SessionState::PopulateKernelCreateInfo()

04bc65d

Merge branch 'edgchen1/memcpy_op_support_for_plugin_eps' into edgchen…

c87eead

…1/memcpy_op_support_for_plugin_eps_fix

check node provider's device instead of hardcoded provider types

9548524

use IsMemcpyNode helper

9bc2533

Merge remote-tracking branch 'origin/main' into edgchen1/memcpy_op_su…

5336604

…pport_for_plugin_eps_fix

skottmckay previously approved these changes Sep 22, 2025

View reviewed changes

Update OperatorKernels.md

5d5b547

adrianlizarraga previously approved these changes Sep 22, 2025

View reviewed changes

update memcpy transformer tests to also set if node provider

525765d

edgchen1 dismissed adrianlizarraga’s stale review via 525765d September 22, 2025 20:18

edgchen1 dismissed skottmckay’s stale review via 525765d September 22, 2025 20:18

edgchen1 requested review from adrianlizarraga and skottmckay September 22, 2025 22:10

move stream null check up and remove TODO comment

9dfdc84

adrianlizarraga approved these changes Sep 23, 2025

View reviewed changes

edgchen1 merged commit 4545732 into main Sep 23, 2025
96 of 97 checks passed

edgchen1 deleted the edgchen1/memcpy_op_support_for_plugin_eps branch September 23, 2025 17:42

adrianlizarraga mentioned this pull request Sep 24, 2025

ORT 1.23.1 cherrypick 1 #26138

Closed

adrianlizarraga mentioned this pull request Sep 24, 2025

ORT 1.23.1 cherrypick 1 [REDO] #26140

Merged

snnn removed the release:1.23.1 label Sep 25, 2025

edgchen1 mentioned this pull request Oct 4, 2025

[EP ABI] Initial support for kernel-based EPs #26206

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

MemcpyFromHost and MemcpyToHost support for plugin EPs #26088

MemcpyFromHost and MemcpyToHost support for plugin EPs #26088

Uh oh!

edgchen1 commented Sep 18, 2025 •

edited

Loading

Uh oh!

github-actions bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

skottmckay left a comment

Uh oh!

Uh oh!

snnn commented Sep 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

MemcpyFromHost and MemcpyToHost support for plugin EPs #26088

MemcpyFromHost and MemcpyToHost support for plugin EPs #26088

Uh oh!

Conversation

edgchen1 commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

skottmckay left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

snnn commented Sep 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

edgchen1 commented Sep 18, 2025 •

edited

Loading