Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

1.16.0 built from source on RHEL8 fails with: undefined symbol: _ZNSt10filesystem7__cxx114path14_M_split_cmptsEv #6047

Open
cjvolzka opened this issue Mar 28, 2024 · 0 comments
Open

1.16.0 built from source on RHEL8 fails with: undefined symbol: _ZNSt10filesystem7__cxx114path14_M_split_cmptsEv #6047

cjvolzka opened this issue Mar 28, 2024 · 0 comments
Labels
bug

Comments

@cjvolzka
Copy link
Contributor

cjvolzka commented Mar 28, 2024 •
edited

Bug Report

Is the issue related to model conversion?

No

Describe the bug

importing ONNX, built from source, on RHEL8 fails with ImportError: /usr/local/lib64/python3.9/site-packages/onnx/onnx_cpp2py_export.cpython-39-x86_64-linux-gnu.so: undefined symbol: _ZNSt10filesystem7__cxx114path14_M_split_cmptsEv

System information

  • OS Platform and Distribution (e.g. Linux Ubuntu 20.04): RHEL 8
  • ONNX version (e.g. 1.13): 1.16.0
  • Python version: 3.9
  • GCC/Compiler version (if compiling from source): 8.5.0
  • CMake version: 3.29.0

Reproduction instructions

  • Notes
    • Issue won't appear on x86 using pip3 install onnx as that installs pre-built wheel files. To replicate on x86 RHEL 8, use pip3 install --no-binary onnx onnx
    • I'm guessing issue will appear on any Linux using GCC less than 9 (based on https://stackoverflow.com/a/33159746).
    • By default, RHEL 8 uses python3.6 which only installs onnx 1.14 via pip. You need to use a newer python so pip will install the latest onnx
pip3 uninstall -y onnx
pip3 install --no-binary onnx onnx
python3 -c "import onnx; print(f'installed onnx version: {onnx.__version__}')"

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib64/python3.9/site-packages/onnx/__init__.py", line 77, in <module>
    from onnx.onnx_cpp2py_export import ONNX_ML
ImportError: /usr/local/lib64/python3.9/site-packages/onnx/onnx_cpp2py_export.cpython-39-x86_64-linux-gnu.so: undefined symbol: _ZNSt10filesystem7__cxx114path14_M_split_cmptsEv

Expected behavior

installed onnx version: 1.16.0
@cjvolzka cjvolzka added the bug label Mar 28, 2024
This was referenced Mar 28, 2024
Merged
Closed
github-merge-queue bot pushed a commit that referenced this issue Apr 12, 2024
@cjvolzka @gramalingam
Prevent crash on import after GCC 8 builds (#6048)
3f24ef3 
### Description
Possible fix for #6047 based on
https://stackoverflow.com/questions/33149878/experimentalfilesystem-linker-error/33159746#33159746

### Motivation and Context
I'm not a cmake expert so there may be a better way to do this. This did
allow me to build ONNX 1.16 on RHEL 8.

---------

Signed-off-by: Charles Volzka <cjvolzka@us.ibm.com>
Signed-off-by: Charles Volzka <42243335+cjvolzka@users.noreply.github.com>
Co-authored-by: Thiago Crepaldi <thiago.crepaldi@microsoft.com>
Co-authored-by: G. Ramalingam <grama@microsoft.com>
gramalingam added a commit to gramalingam/onnx that referenced this issue Apr 12, 2024
@cjvolzka @gramalingam
Prevent crash on import after GCC 8 builds (onnx#6048)
4940167 
### Description
Possible fix for onnx#6047 based on
https://stackoverflow.com/questions/33149878/experimentalfilesystem-linker-error/33159746#33159746

### Motivation and Context
I'm not a cmake expert so there may be a better way to do this. This did
allow me to build ONNX 1.16 on RHEL 8.

---------

Signed-off-by: Charles Volzka <cjvolzka@us.ibm.com>
Signed-off-by: Charles Volzka <42243335+cjvolzka@users.noreply.github.com>
Co-authored-by: Thiago Crepaldi <thiago.crepaldi@microsoft.com>
Co-authored-by: G. Ramalingam <grama@microsoft.com>
Signed-off-by: Ganesan Ramalingam <grama@microsoft.com>
gramalingam added a commit to gramalingam/onnx that referenced this issue Apr 12, 2024
@cjvolzka @gramalingam
Prevent crash on import after GCC 8 builds (onnx#6048)
0c387b9 
### Description
Possible fix for onnx#6047 based on
https://stackoverflow.com/questions/33149878/experimentalfilesystem-linker-error/33159746#33159746

### Motivation and Context
I'm not a cmake expert so there may be a better way to do this. This did
allow me to build ONNX 1.16 on RHEL 8.

---------

Signed-off-by: Charles Volzka <cjvolzka@us.ibm.com>
Signed-off-by: Charles Volzka <42243335+cjvolzka@users.noreply.github.com>
Co-authored-by: Thiago Crepaldi <thiago.crepaldi@microsoft.com>
Co-authored-by: G. Ramalingam <grama@microsoft.com>
Signed-off-by: Ganesan Ramalingam <grama@microsoft.com>
cjvolzka added a commit that referenced this issue Apr 29, 2024
@cjvolzka
Prevent crash on import after GCC 8 builds (#6048)
77b5255 
### Description
Possible fix for #6047 based on
https://stackoverflow.com/questions/33149878/experimentalfilesystem-linker-error/33159746#33159746

### Motivation and Context
I'm not a cmake expert so there may be a better way to do this. This did
allow me to build ONNX 1.16 on RHEL 8.

---------

Signed-off-by: Charles Volzka <cjvolzka@us.ibm.com>
Signed-off-by: Charles Volzka <42243335+cjvolzka@users.noreply.github.com>
Co-authored-by: Thiago Crepaldi <thiago.crepaldi@microsoft.com>
Co-authored-by: G. Ramalingam <grama@microsoft.com>
(cherry picked from commit 3f24ef3)
@snnn snnn mentioned this issue May 24, 2024
Merged
snnn added a commit to microsoft/onnxruntime that referenced this issue May 24, 2024
@snnn
Remove manylinux build scripts from python packaging pipeline (#20786)
535a030 
### Description
Use a common set of prebuilt manylinux base images to build the
packages, to avoid building the manylinux part again and again. The base
images can be used in GenAI and other projects too.
This PR also updates the GCC version for inference python CUDA11/CUDA12
builds from 8 to 11. Later on I will update all other CUDA pipelines to
use GCC 11, to avoid the issue described in
onnx/onnx#6047 and
microsoft/onnxruntime-genai#257 .

### Motivation and Context
To extract the common part as a reusable build infra among different
ONNX Runtime projects.
@snnn snnn mentioned this issue Jun 1, 2024
Merged
snnn added a commit to microsoft/onnxruntime that referenced this issue Jun 3, 2024
@snnn
Upgrade GCC and remove the dependency on GCC8's experimental std::fil…
d13cabf 
…esystem implementation (#20893)

### Description
This PR upgrades CUDA 11 build pipelines' GCC version from 8 to 11. 

### Motivation and Context

GCC8 has an experimental std::filesystem implementation which is not ABI
compatible with the formal one in later GCC releases. It didn't cause
trouble for us, however, ONNX community has encountered this issue much.
For example, onnx/onnx#6047 . So this PR
increases the minimum supported GCC version from 8 to 9, and removes the
references to GCC's "stdc++fs" library. Please note we compile our code
on RHEL8 and RHEL8's libstdc++ doesn't have the fs library, which means
the binaries in ONNX Runtime's official packages always static link to
the fs library. It is just a matter of which version of the library, an
experimental one or a more mature one. And it is an implementation
detail that is not visible from outside. Anyway, a newer GCC is better.
It will give us the chance to use many C++20 features.

#### Why we were using GCC 8?
It is because all our Linux packages were built on RHEL8 or its
equivalents. The default GCC version in RHEL8 is 8. RHEL also provides
additional GCC versions from RH devtoolset. UBI8 is the abbreviation of
Red Hat Universal Base Image 8, which is the containerized RHEL8. UBI8
is free, which means it doesn't require a subscription(while RHEL does).
The only devtoolset that UBI8 provides is GCC 12, which is too new for
being used with CUDA 11.8. And our CUDA 11.8's build env is a docker
image from Nvidia that is based on UBI8.
#### How the problem is solved
Almalinux is an alternative to RHEL. Almalinux 8 provides GCC 11. And
the CUDA 11.8 docker image from Nvidia is open source, which means we
can rebuild the image based on Almalinux 8 to get GCC 11. I've done
this, but I cannot republish the new image due to various complicated
license restrictions. Therefore I put them at an internal location in
onnxruntimebuildcache.azurecr.io.
gramalingam added a commit to gramalingam/onnx that referenced this issue Jun 6, 2024
@cjvolzka @gramalingam
Prevent crash on import after GCC 8 builds (onnx#6048)
eb1fd58 
### Description
Possible fix for onnx#6047 based on
https://stackoverflow.com/questions/33149878/experimentalfilesystem-linker-error/33159746#33159746

### Motivation and Context
I'm not a cmake expert so there may be a better way to do this. This did
allow me to build ONNX 1.16 on RHEL 8.

---------

Signed-off-by: Charles Volzka <cjvolzka@us.ibm.com>
Signed-off-by: Charles Volzka <42243335+cjvolzka@users.noreply.github.com>
Co-authored-by: Thiago Crepaldi <thiago.crepaldi@microsoft.com>
Co-authored-by: G. Ramalingam <grama@microsoft.com>
Signed-off-by: Ganesan Ramalingam <grama@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cjvolzka