tf.load_op_library unable to load manylinux2010 repaired custom ops #31807
Description
System information
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No -- using https://github.com/tensorflow/custom-op (But it breaks for addons too)
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu16.04
- TensorFlow installed from (source or binary): Binary
- TensorFlow version (use command below): tf-nightly & tf-nighty-2.0-preview
Describe the current behavior
Currently when I build a custom op in the tensorflow/tensorflow:custom-op-ubuntu16 docker image using the defined steps I get an install-able pip package tensorflow_zero_out-0.0.1-cp27-cp27mu-linux_x86_64.whl
This works fine, however if I repair that wheel to be manylinux2010 compliant, then tf.load_op_library will fail to find the custom-op.
python -c "import tensorflow as tf; print(dir(tf.load_op_library('manylinux/tensorflow_zero_out/python/ops/_zero_out_ops.so')))"
['LIB_HANDLE', 'OP_LIST', 'ZeroOut', '_IS_TENSORFLOW_PLUGIN',
'_InitOpDefLibrary', '__builtins__', '__doc__', '__name__', '__package__',
'_collections', '_common_shapes', '_context', '_core', '_dispatch', '_doc_controls',
'_dtypes', '_errors', '_execute', '_kwarg_only', '_op_def_lib', '_op_def_library',
'_op_def_pb2', '_op_def_registry', '_ops', '_pywrap_tensorflow', '_six',
'_tensor_shape', 'deprecated_endpoints', 'tf_export', 'zero_out',
'zero_out_eager_fallback']
python -c "import tensorflow as tf;print(dir(tf.load_op_library('manylinux2010/tensorflow_zero_out/python/ops/_zero_out_ops.so')))"
['LIB_HANDLE', 'OP_LIST', '_IS_TENSORFLOW_PLUGIN',
'_InitOpDefLibrary', '__builtins__', '__doc__', '__name__', '__package__',
'_collections', '_common_shapes', '_context', '_core',
'_dispatch', '_doc_controls', '_dtypes', '_errors', '_execute', '_kwarg_only',
'_op_def_lib', '_op_def_library', '_op_def_pb2', '_op_def_registry', '_ops',
'_pywrap_tensorflow', '_six', '_tensor_shape', 'deprecated_endpoints', 'tf_export']
Notice 'zero_out' & 'zero_out_eager_fallback' are not found in the loaded library for manylinux2010
Code to reproduce the issue
git clone https://github.com/tensorflow/custom-op.git && cd custom-op
docker run -it --rm -v ${PWD}:/workspace -w /workspace tensorflow/tensorflow:custom-op-ubuntu16 /bin/bash
pip install tf-nightly
./configure.sh
bazel build build_pip_pkg
bazel-bin/build_pip_pkg artifacts
# Installed auditwheel is too old for manylinux2010
pip3 install --upgrade auditwheel
# Libtensorflow framework needs to be on LD path
export LD_LIBRARY_PATH="/usr/local/lib/python2.7/dist-packages/tensorflow_core"
# Repair logs look more or less okay
auditwheel -v repair --plat manylinux2010_x86_64 artifacts/tensorflow_zero_out-0.0.1-cp27-cp27mu-linux_x86_64.whl &> repair.txt
Other info / logs
Here are the auditwheel repair logs:
repair.txt
Here are the readelf inspections of the so files:
readelf.txt
readelf-manylinux2010.txt
Here are the so files:
so-files.zip
--------------------------EDIT--------------------
Here are the extracted whl directories which will work with the python tf.load_op_library commands from above. (Manylinux2010 repair makes it so the custom op depends on a newly copied libtensorflow_framework.so which is part of the new whl):
custom-op-dirs.zip