Reshape feature implementation #573

jatinwadhwa921 · 2025-02-11T05:25:46Z

Reshape feature implementation, This feature will help you set lower and upper bound for ov tensors only for NPU. Command used to run the feature -

onnxruntime_perf_test.exe -v -e openvino -m times -r 1 -i "device_type|NPU reshape_input|data[1,3,60,80..120]" <model_path>

sfatimar · 2025-02-28T09:47:50Z

@jatinwadhwa921 please update this branch

jatinwadhwa921 · 2025-02-28T09:49:48Z

@jatinwadhwa921 please update this branch

sure, i will rebase this branch again with latest ovep-develop

sfatimar · 2025-03-04T04:14:12Z

onnxruntime/core/providers/openvino/openvino_provider_factory.cc

@@ -236,6 +236,97 @@ struct OpenVINO_Provider : Provider {

    pi.precision = ParsePrecision(provider_options, pi.device_type, "precision");

+    if (provider_options.contains("reshape_input") && pi.device_type == "NPU") {


@jatinwadhwa921 exactly what are we trying to do here

I am not comfortable leaving so much parsing in main functions can we create a file parse_utils.cc and dump all parsing functions there

will move the parser functions at the time of rebasing

sfatimar · 2025-03-04T04:14:38Z

@preetha-intel @ankitm3k can you please review this PR

sfatimar · 2025-03-10T07:49:50Z

I would expect all parsing functions inside openvino_provider_factory to move to parse utils.

jatinwadhwa921 · 2025-04-28T05:53:54Z

Design Document for reshape_input.docx
Attaching the design document for this feature

sfatimar · 2025-04-29T07:27:23Z

onnxruntime/core/providers/openvino/backend_manager.cc

-  // Save the indexes of graph inputs among fused_node's inputDefs
-  // (which also contains initializers).
+  if (!session_context_.shape.empty()) {
+    ValidateInputShapes(session_context_.shape, subgraph.GetInputs());


Why is this added here

preetha-intel · 2025-04-29T07:27:52Z

onnxruntime/core/providers/openvino/backend_manager.cc

  for (uint32_t index = 0; const auto& node : subgraph.GetInputs()) {
+   if(subgraph.GetGraph().GetConsumerNodes(node->Name()).size()==0)


Add a comment on to why this is required. Are there dangling inputs ?

sfatimar · 2025-04-29T07:28:20Z

onnxruntime/core/providers/openvino/backend_manager.cc

  for (uint32_t index = 0; const auto& node : subgraph.GetInputs()) {
+   if(subgraph.GetGraph().GetConsumerNodes(node->Name()).size()==0)


What is this part of code doing

sfatimar · 2025-04-29T07:29:24Z

onnxruntime/core/providers/openvino/backend_manager.cc

@@ -100,7 +108,7 @@ BackendManager::BackendManager(SessionContext& session_context,
    }
  }

-  if (ModelHasSymbolicInputDims(subgraph)) {
+  if (ModelHasSymbolicInputDims(subgraph) && session_context_.shape.empty()) {


Is shapeempty checking for upper bound, lower bound ?

I think this portion of code needs to be rewritten... to be device agnostic and to have one approach for dynamism

sfatimar · 2025-04-29T07:32:19Z

onnxruntime/core/providers/openvino/backend_manager.h

@@ -39,6 +39,8 @@ class BackendManager {

  bool ModelHasSymbolicInputDims(const onnxruntime::GraphViewer& subgraph) const;
  bool ModelHasBatchedInputs(const ONNX_NAMESPACE::ModelProto& model_proto) const;
+  void ValidateInputShapes(const shape_t& shape,


ValidateInputShapes do not have a string argument in declaration but in definition there is one. Is it by design.

sfatimar · 2025-04-29T07:33:46Z

onnxruntime/core/providers/openvino/backend_utils.cc

@@ -146,6 +146,11 @@ CreateOVModel(const std::string model,
  try {
    auto ov_model = OVCore::Get()->ReadModel(model, session_context.onnx_model_path_name.string());

+    if (!session_context.shape.empty()) {
+      LOGS_DEFAULT(INFO) << log_tag << "Reshaping the ov tensor to specified shape";
+      ov_model->reshape(session_context.shape);


This converts the model to static shape ?

preetha-intel · 2025-04-29T07:35:08Z

onnxruntime/core/providers/openvino/backend_utils.cc

@@ -146,6 +146,11 @@ CreateOVModel(const std::string model,
  try {
    auto ov_model = OVCore::Get()->ReadModel(model, session_context.onnx_model_path_name.string());

+    if (!session_context.shape.empty()) {
+      LOGS_DEFAULT(INFO) << log_tag << "Reshaping the ov tensor to specified shape";


" Reshape the model inputs to specified shape "

sfatimar · 2025-04-29T07:44:58Z

onnxruntime/core/providers/openvino/backends/basic_backend.cc

@@ -96,6 +97,7 @@ BasicBackend::BasicBackend(std::unique_ptr<ONNX_NAMESPACE::ModelProto>& model_pr
    } else if (!session_context_.has_external_weights &&
               !subgraph_context_.has_dynamic_input_shape &&
               !session_context_.so_context_enable &&
+               session_context.shape.empty() &&


As I see the model still has dynamic shape so this should not be here.

And by keeping this here we are saying that we wont use unified compile model API for upper bound and lower bound models... which will impact FIL.

sfatimar · 2025-04-29T07:49:33Z

onnxruntime/core/providers/openvino/backends/basic_backend.cc

-            ov_tensor_data.tensor_ptr = std::make_shared<ov::Tensor>(input.get_element_type(), input.get_shape(),
-                                                                     const_cast<void*>(tensor.GetTensorRawData()));
-
+            if (!session_context_.shape.empty()) {


I think we should change the logic in top most braces:

if (subgraph_context_.has_dynamic_input_shape &&
!session_context_.disable_dynamic_shapes ) {

sfatimar · 2025-04-29T07:52:14Z

onnxruntime/core/providers/openvino/backends/basic_backend.cc

+                                                                       const_cast<void*>(tensor.GetTensorRawData()));
+            } else {
+              ov_tensor_data.tensor_ptr = std::make_shared<ov::Tensor>(input.get_element_type(), input.get_shape(),
+                                                                       const_cast<void*>(tensor.GetTensorRawData()));


I think the whole logic of startasyncinference needs to be rewritten

sfatimar · 2025-04-29T07:54:29Z

onnxruntime/core/providers/openvino/backends/basic_backend.cc

@@ -434,6 +447,10 @@ void BasicBackend::StartAsyncInference(Ort::KernelContext& context, OVInferReque
      }
    }  // Loop subgraph original input names

+    if (!session_context_.shape.empty()) {


This infer should not be here.

Infer should only be called from Concrete backend or dynamic backend... this is breaking existing design

sfatimar · 2025-04-29T08:03:15Z

I think should be removed:

sfatimar · 2025-04-29T08:03:25Z

// Always true for NPU plugin or when passed .
if (pi.device_type.find("NPU") != std::string::npos) {
pi.disable_dynamic_shapes = true;
}

sfatimar · 2025-04-29T09:00:31Z

I would logic for dynamic shapes to be correctly documented than putting if else conditions randomly. I think instead of adding shapes information everywhere and checking for CPU, GPU please just rely on flags disable_dynamic_shapes and has_dynamic_input_shape for dynamic shapes . Currently since NPU does not support dynamic shapes logic should be:
First logic for provider options flags based on device:

If device type is NPU and reshape is not present enable dynamic backend. (disable dynamics shape=true)
If device type is NPU and reshape is present enable dynamic inputs. (disable_dynamic shape = false)
Uniform logic in subsequent code
If model input is dynamic .and disable_dynamic shape is true (dynamic backend is enabled) go for dynamic backend. Reshape parameters have no effect.
If model input is dynamic and disable_dynamic_backend is false(dynamic backend is disabled) go for concrete backend and dynamic inputs.
If shape parameters are present apply partial shape/reshape to model and inputs (Read Model/Compile Model or Unified Compile Model -- Preetha/Jatin to give feedback here)

ExportCompiledBlobAsEPCtxNode
If dynamic backend is disabled and model is dynamic go for export ep context model...

OrestChura · 2025-05-02T17:35:16Z

onnxruntime/core/providers/openvino/contexts.h

@@ -79,6 +80,7 @@ struct ProviderInfo {
  uint32_t num_of_threads{0};              // [num_of_threads]: Overrides the accelerator default value of
                                           // number of threads with this value at runtime.
  config_t load_config{};                  // JSON config map to load custom OV parameters.
+  shape_t shape{};                         // Used for reshaping ov tensors to a particular lower and upper bound


would it be possible to name this variable as the EP-specific option named - "reshape_input", or at least mention that name in the comment?
As this is the only place where OVEP-specific options are listed in comprehensible format, so we're referring to these names/comments.

Could also the description of this be added to https://github.com/intel/onnxruntime/blob/master/onnxruntime/test/perftest/command_args_parser.cc please?

Btw is there a documentation dedicated to OVEP-specific runtime options?

there's also the "valid_provider_keys" field (added recently, probably)

onnxruntime/onnxruntime/core/providers/openvino/contexts.h

Line 106 in 89965af

const std::unordered_set<std::string> valid_provider_keys = {"device_type", "device_id", "device_luid", "cache_dir", "precision",

could this be added there too?

jatinwadhwa921 requested a review from preetha-intel February 11, 2025 05:25

sfatimar force-pushed the ovep-develop branch from 059f7c1 to f84614c Compare February 11, 2025 11:39

ankitm3k force-pushed the ovep-develop branch 3 times, most recently from b66301b to e93f0b0 Compare February 17, 2025 09:28

jatinwadhwa921 force-pushed the jatin_latest_reshape_refactor branch 2 times, most recently from 53278fe to be37fd9 Compare February 17, 2025 17:23

jatinwadhwa921 marked this pull request as ready for review February 17, 2025 17:28

jatinwadhwa921 requested review from ankitm3k, saurabhkale17 and sfatimar February 18, 2025 04:50

ankitm3k force-pushed the ovep-develop branch 2 times, most recently from 42d6f14 to e85411a Compare February 24, 2025 13:21

Reshape feature implementation

ca2bc91

jatinwadhwa921 force-pushed the jatin_latest_reshape_refactor branch from be37fd9 to ca2bc91 Compare March 3, 2025 10:13

sfatimar reviewed Mar 4, 2025

View reviewed changes

sfatimar reviewed Apr 29, 2025

View reviewed changes

preetha-intel reviewed Apr 29, 2025

View reviewed changes

sfatimar reviewed Apr 29, 2025

View reviewed changes

preetha-intel reviewed Apr 29, 2025

View reviewed changes

sfatimar reviewed Apr 29, 2025

View reviewed changes

OrestChura reviewed May 2, 2025

View reviewed changes

		@@ -236,6 +236,97 @@ struct OpenVINO_Provider : Provider {

		pi.precision = ParsePrecision(provider_options, pi.device_type, "precision");

		if (provider_options.contains("reshape_input") && pi.device_type == "NPU") {

		for (uint32_t index = 0; const auto& node : subgraph.GetInputs()) {
		if(subgraph.GetGraph().GetConsumerNodes(node->Name()).size()==0)

Reshape feature implementation #573

Are you sure you want to change the base?

Reshape feature implementation #573

Uh oh!

Conversation

jatinwadhwa921 commented Feb 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sfatimar commented Feb 28, 2025

Uh oh!

jatinwadhwa921 commented Feb 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfatimar commented Mar 4, 2025

Uh oh!

sfatimar commented Mar 10, 2025

Uh oh!

jatinwadhwa921 commented Apr 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfatimar commented Apr 29, 2025

Uh oh!

sfatimar commented Apr 29, 2025

Uh oh!

sfatimar commented Apr 29, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jatinwadhwa921 commented Feb 11, 2025 •

edited

Loading