Thanks a lot for the PR, it's an issue I've had with large models but did not look into it, this will surely help!

The documentation is not available anymore as the PR was closed or merged.

@VikParuchuri This step str(tensor_data).encode("utf-8") is quite slow. I wonder if there is anything better we could hash than this? I suspect for large models this will be prohibitively slow.

Hashing simply tensor_data does not seem to work, getting object supporting the buffer API required. This is kind of surprising as it's of type <class 'bytes'>, and what you get after the encode is also a byte? Any idea @michaelbenayoun?

The reason we put that in a tuple is that some initializers have names as onnx::MatMul_3741, that may be different between the two graphs. So we need to look into the actual data to check whether they are the same.

Hi @VikParuchuri,

That's great thanks!

This is due to the tensor_data in _find_duplicate_initializers being converted from bytes to a tuple. I assume there is a good reason to convert it to a tuple (maybe for comparison across dtypes).

I actually do not remember why make tensor_data a tuple, have you tried removing this? I tried and it seems to be working, at least for the one model I tried.

For reference: https://github.com/huggingface/optimum/pull/587/files#r1152974762. I think it's safe to keep as a tuple + use to_array for now.

@fxmarty I am not talking about the tuple in the dictionary but tensor_data.

@fxmarty Your method seems to work! I updated the PR.

@VikParuchuri Great! Could you do pip install -e .[quality] and make style?

Done