ENH:Distutils Remove debugging symbols by default #16110

seiko2plus · 2020-04-29T19:42:15Z

Debug info flags are always set by the default which increases
the binary size to over 600%, in case of the debug info
is required for backtraces, the user has to rebuild NumPy
with build option --debug.

Shared object sizes with debug flags

du */*.so
396	core/_multiarray_tests.cpython-37m-x86_64-linux-gnu.so
19144	core/_multiarray_umath.cpython-37m-x86_64-linux-gnu.so
48	core/_operand_flag_tests.cpython-37m-x86_64-linux-gnu.so
256	core/_rational_tests.cpython-37m-x86_64-linux-gnu.so
52	core/_struct_ufunc_tests.cpython-37m-x86_64-linux-gnu.so
104	core/_umath_tests.cpython-37m-x86_64-linux-gnu.so
360	fft/_pocketfft_internal.cpython-37m-x86_64-linux-gnu.so
4928	linalg/lapack_lite.cpython-37m-x86_64-linux-gnu.so
5536	linalg/_umath_linalg.cpython-37m-x86_64-linux-gnu.so
904	random/bit_generator.cpython-37m-x86_64-linux-gnu.so
2028	random/_bounded_integers.cpython-37m-x86_64-linux-gnu.so
1504	random/_common.cpython-37m-x86_64-linux-gnu.so
2940	random/_generator.cpython-37m-x86_64-linux-gnu.so
556	random/_mt19937.cpython-37m-x86_64-linux-gnu.so
2372	random/mtrand.cpython-37m-x86_64-linux-gnu.so
376	random/_pcg64.cpython-37m-x86_64-linux-gnu.so
540	random/_philox.cpython-37m-x86_64-linux-gnu.so
304	random/_sfc64.cpython-37m-x86_64-linux-gnu.so

Shared object sizes "without" debug flags

du */*.so
136	core/_multiarray_tests.cpython-37m-x86_64-linux-gnu.so
3564	core/_multiarray_umath.cpython-37m-x86_64-linux-gnu.so
20	core/_operand_flag_tests.cpython-37m-x86_64-linux-gnu.so
52	core/_rational_tests.cpython-37m-x86_64-linux-gnu.so
20	core/_struct_ufunc_tests.cpython-37m-x86_64-linux-gnu.so
28	core/_umath_tests.cpython-37m-x86_64-linux-gnu.so
76	fft/_pocketfft_internal.cpython-37m-x86_64-linux-gnu.so
1792	linalg/lapack_lite.cpython-37m-x86_64-linux-gnu.so
1924	linalg/_umath_linalg.cpython-37m-x86_64-linux-gnu.so
224	random/bit_generator.cpython-37m-x86_64-linux-gnu.so
388	random/_bounded_integers.cpython-37m-x86_64-linux-gnu.so
268	random/_common.cpython-37m-x86_64-linux-gnu.so
864	random/_generator.cpython-37m-x86_64-linux-gnu.so
140	random/_mt19937.cpython-37m-x86_64-linux-gnu.so
748	random/mtrand.cpython-37m-x86_64-linux-gnu.so
92	random/_pcg64.cpython-37m-x86_64-linux-gnu.so
116	random/_philox.cpython-37m-x86_64-linux-gnu.so
76	random/_sfc64.cpython-37m-x86_64-linux-gnu.so

Umath extension binary info via Bloaty with debug flags

bloaty core/_multiarray_umath.cpython-37m-x86_64-linux-gnu.so
    FILE SIZE        VM SIZE    
 --------------  -------------- 
  45.0%  9.39Mi   0.0%       0    .debug_loc
  22.0%  4.58Mi   0.0%       0    .debug_info
  11.4%  2.38Mi  71.8%  2.38Mi    .text
   9.7%  2.03Mi   0.0%       0    .debug_line
   5.3%  1.10Mi   0.0%       0    .debug_ranges
   1.2%   263Ki   7.8%   263Ki    .eh_frame
   1.1%   233Ki   6.9%   233Ki    .rodata
   0.9%   182Ki   0.0%       0    .symtab
   0.8%   174Ki   0.0%       0    .debug_str
   0.0%       0   3.9%   130Ki    .bss
   0.6%   117Ki   0.0%       0    .strtab
   0.5%   110Ki   3.3%   110Ki    .data
   0.5%   107Ki   0.0%       0    .debug_abbrev
   0.5%   100Ki   3.0%   100Ki    .rela.dyn
   0.2%  50.0Ki   1.5%  49.9Ki    .eh_frame_hdr
   0.1%  22.1Ki   0.6%  22.0Ki    .dynsym
   0.1%  22.0Ki   0.4%  12.5Ki    [21 Others]
   0.1%  10.9Ki   0.3%  10.9Ki    .dynstr
   0.0%  9.46Ki   0.3%  9.40Ki    .rela.plt
   0.0%  6.34Ki   0.2%  6.28Ki    .plt
   0.0%  5.94Ki   0.2%  5.88Ki    .gnu.hash
 100.0%  20.9Mi 100.0%  3.31Mi    TOTAL

Umath extension binary info via Bloaty "without" debug flags

bloaty core/_multiarray_umath.cpython-37m-x86_64-linux-gnu.so
    FILE SIZE        VM SIZE    
 --------------  -------------- 
  68.3%  2.38Mi  71.8%  2.38Mi    .text
   7.4%   263Ki   7.8%   263Ki    .eh_frame
   6.6%   233Ki   6.9%   233Ki    .rodata
   5.1%   182Ki   0.0%       0    .symtab
   0.0%       0   3.9%   130Ki    .bss
   3.3%   117Ki   0.0%       0    .strtab
   3.1%   110Ki   3.3%   110Ki    .data
   2.8%   100Ki   3.0%   100Ki    .rela.dyn
   1.4%  50.0Ki   1.5%  49.9Ki    .eh_frame_hdr
   0.6%  22.1Ki   0.6%  22.0Ki    .dynsym
   0.3%  10.9Ki   0.3%  10.9Ki    .dynstr
   0.3%  9.46Ki   0.3%  9.40Ki    .rela.plt
   0.2%  6.34Ki   0.2%  6.28Ki    .plt
   0.2%  5.94Ki   0.2%  5.88Ki    .gnu.hash
   0.2%  5.40Ki   0.2%  5.15Ki    .data.rel.ro
   0.1%  4.56Ki   0.0%       0    [Unmapped]
   0.1%  3.22Ki   0.1%  3.16Ki    .got.plt
   0.1%  2.03Ki   0.0%  1.12Ki    [14 Others]
   0.1%  1.90Ki   0.1%  1.83Ki    .gnu.version
   0.0%     712   0.0%     648    .got
   0.0%     632   0.0%     632    [LOAD #0 [R]]
 100.0%  3.48Mi 100.0%  3.31Mi    TOTAL

seiko2plus · 2020-04-29T20:41:46Z

runtests.py

what the point of using -ggdb over -g?

Probably gdb format was more useful than OS native format in the past. However it's 2020 now, and assuming everyone uses GCC seems a little outdated.

make sense thanks!, I kept it untouched, I will try to release a cleanup patch to beautify build logs and remove such cases like this one.

rgommers · 2020-04-30T05:32:18Z

Sounds like a good idea to me in principle. Note that our wheels do get stripped (MacPython/numpy-wheels#19), however there's many ways to build NumPy and debug info is only needed rarely so non-debug as default makes sense.

rgommers

Thanks @seiko2plus! I like the goal of this, not quite sure about the way of filtering. Also, Fortran compilers should probably be treated the same way as C/C++ compilers.

rgommers · 2020-04-30T05:41:42Z

numpy/distutils/ccompiler.py

It seems to me like trying to strip out flags that are set already may be confusing. This would also filter out flags explicitly set in for example CFLAGS right?

Oh my bad, I fixed it by execluding debug flags only if they weren't part of env vars.

Debug info flags are always set by the default which increases the binary size to over 600%, in case of the debug info is required for backtraces, the user has to rebuild NumPy with build option --debug.

seiko2plus · 2020-04-30T18:47:20Z

@rgommers,

Sounds like a good idea to me in principle. Note that our wheels do get stripped (MacPython/numpy-wheels#19),

But I wonder why the wheel package that I fetched through PIP still has debug info?

however there's many ways to build NumPy and debug info is only needed rarely so non-debug as default makes sense.

It also makes --debug option is useless since debug flags always been set by sysconfig vars

seiko2plus · 2020-04-30T18:52:24Z

@rgommers

Thanks @seiko2plus! I like the goal of this, not quite sure about the way of filtering. Also, Fortran compilers should probably be treated the same way as C/C++ compilers.

I update it doc string, to add more clarification, I'm not sure yet about Fortran I will check it.

mattip · 2020-04-30T21:03:53Z

But I wonder why the wheel package that I fetched through PIP still has debug info?

Which wheel package are you using?

seiko2plus · 2020-04-30T21:05:41Z

@mattip, numpy-1.18.3-cp37-cp37m-manylinux1_x86_64.whl

du site-packages/numpy/core/_multiarray_umath.cpython-37m-x86_64-linux-gnu.so
20968	site-packages/numpy/core/_multiarray_umath.cpython-37m-x86_64-linux-gnu.so

bloaty site-packages/numpy/core/_multiarray_umath.cpython-37m-x86_64-linux-gnu.so
    FILE SIZE        VM SIZE    
 --------------  -------------- 
  51.6%  10.6Mi   0.0%       0    .debug_loc
  18.0%  3.69Mi   0.0%       0    .debug_info
  14.3%  2.93Mi  74.9%  2.93Mi    .text
   5.9%  1.21Mi   0.0%       0    .debug_ranges
   3.2%   673Ki   0.0%       0    .debug_line
   1.3%   281Ki   7.0%   281Ki    .eh_frame
   1.1%   228Ki   5.7%   228Ki    .rodata
   0.8%   178Ki   0.0%       0    .symtab
   0.8%   163Ki   0.0%       0    .debug_str
   0.7%   154Ki   0.0%       0    .strtab
   0.0%       0   3.3%   131Ki    .bss
   0.5%   110Ki   2.8%   110Ki    .data
   0.5%   101Ki   2.5%   101Ki    .rela.dyn
   0.4%  87.0Ki   0.0%       0    .debug_abbrev
   0.2%  49.1Ki   1.2%  49.0Ki    .eh_frame_hdr
   0.2%  35.4Ki   0.7%  27.4Ki    [22 Others]
   0.1%  22.5Ki   0.6%  22.5Ki    .dynsym
   0.1%  15.0Ki   0.4%  15.0Ki    [LOAD #0 [RX]]
   0.1%  14.9Ki   0.4%  14.8Ki    .rela.plt
   0.1%  11.2Ki   0.3%  11.1Ki    .dynstr
   0.1%  11.1Ki   0.3%  11.1Ki    [LOAD #6 [RW]]
 100.0%  20.5Mi 100.0%  3.91Mi    TOTAL

mattip · 2020-04-30T21:11:23Z

I think the CFLAGS that are meant to strip the binaries are being reset by the env_vars.sh script in numpy-wheels

mattip · 2020-04-30T21:24:31Z

As far as I can tell, these flags are coming from the host python: sysconfig.get_config_vars('CFLAGS')

I don't think this is a numpy-specific problem, it is due to the way Ubuntu/Debian build python. We should fix our wheel building, but I am not sure we need to interfere with the flags here.

seiko2plus · 2020-04-30T21:41:21Z

@mattip, yes flags come from sysconfig var CFLAGS and also OPT but since we have a custom ccompiler, then we can consider them as optional flags. also, handle flags through env CFLAGS not a clean way it supposed to be an external option for the end-user.

mattip · 2020-04-30T21:50:09Z

xref MacPython/numpy-wheels#82

charris · 2020-05-12T20:11:59Z

Do we want to make stripping a default or leave that choice to downstream projects?

seiko2plus · 2020-05-12T21:15:38Z

@charris, we should leave them the choice to enable debugging symbols via CFLAGS or through --debug not the opposite. that been part of the culture industry for along time.

charris · 2020-05-12T21:52:21Z

If we are changing the behavior then we need a release note telling folks that debug versions need to be specified and how to do it. Maybe in the "Changes" section.

mattip · 2020-05-13T05:52:08Z

I am -0.5 on accepting this. It seems to be a local fix for a python-wide problem. I think a better place to change flags would be upstream: perhaps in distutils (or setuptools) or
to convince CPython to remove the flag in sysconfig.

Note the problem of wheel size was fixed in MacPython/numpy-wheels#87.

seiko2plus · 2020-05-13T22:18:52Z

@mattip, @rgommers, @charris, wherever your decision is, from now on any reports about the size from my side that related to my upcoming patches, it will assume that the debugging symbols aren't combined with NumPy shared objects.

rgommers · 2020-05-13T22:33:20Z

That sounds good @seiko2plus, thanks!

seberg · 2020-05-14T22:04:29Z

IIRC we decided to not put this in, since the wheels are where the size really matters and those are being stripped now. Thanks @seiko2plus. (please reopen if I misremembered.)

seiko2plus · 2020-05-14T22:12:04Z

@seberg, no problem, fully respect it. thanks.

seiko2plus marked this pull request as ready for review April 29, 2020 20:07

seiko2plus force-pushed the fix_nodebug_build branch from b25224d to 4373d2a Compare April 29, 2020 20:32

seiko2plus commented Apr 29, 2020

View reviewed changes

rgommers added 01 - Enhancement component: numpy.distutils labels Apr 30, 2020

rgommers reviewed Apr 30, 2020

View reviewed changes

ENH:Distutils Remove debugging symbols by default

a20bb72

Debug info flags are always set by the default which increases the binary size to over 600%, in case of the debug info is required for backtraces, the user has to rebuild NumPy with build option --debug.

seiko2plus force-pushed the fix_nodebug_build branch from 4373d2a to a20bb72 Compare April 30, 2020 18:37

YannickJadoul mentioned this pull request Apr 30, 2020

Strip debug symbols of wheels pypa/cibuildwheel#331

Open

This was referenced May 1, 2020

ENH: enable multi-platform SIMD compiler optimizations #13516

Merged

Fix stripping binaries from debugging symbols MacPython/numpy-wheels#87

Merged

seberg closed this May 14, 2020

jameslamb mentioned this pull request Sep 7, 2022

[new check] wheel contains compiled objects/libraries with debug symbols jameslamb/pydistcheck#54

Open

6 tasks

This was referenced Mar 9, 2023

[python] Linux wheels contain debug symbols catboost/catboost#2318

Open

ENH: strip debug symbols from macOS wheels pandas-dev/pandas#51900

Closed

Uh oh!

ENH:Distutils Remove debugging symbols by default #16110

ENH:Distutils Remove debugging symbols by default #16110

Uh oh!

Conversation

seiko2plus commented Apr 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seiko2plus Apr 29, 2020

Choose a reason for hiding this comment

Uh oh!

rgommers Apr 30, 2020

Choose a reason for hiding this comment

Uh oh!

seiko2plus Apr 30, 2020

Choose a reason for hiding this comment

Uh oh!

rgommers commented Apr 30, 2020

Uh oh!

rgommers left a comment

Choose a reason for hiding this comment

Uh oh!

rgommers Apr 30, 2020

Choose a reason for hiding this comment

Uh oh!

seiko2plus Apr 30, 2020

Choose a reason for hiding this comment

Uh oh!

seiko2plus commented Apr 30, 2020

Uh oh!

seiko2plus commented Apr 30, 2020

Uh oh!

mattip commented Apr 30, 2020

Uh oh!

seiko2plus commented Apr 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mattip commented Apr 30, 2020

Uh oh!

mattip commented Apr 30, 2020

Uh oh!

seiko2plus commented Apr 30, 2020

Uh oh!

mattip commented Apr 30, 2020

Uh oh!

charris commented May 12, 2020

Uh oh!

seiko2plus commented May 12, 2020

Uh oh!

charris commented May 12, 2020

Uh oh!

mattip commented May 13, 2020

Uh oh!

seiko2plus commented May 13, 2020

Uh oh!

rgommers commented May 13, 2020

Uh oh!

seberg commented May 14, 2020

Uh oh!

seiko2plus commented May 14, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

seiko2plus commented Apr 29, 2020 •

edited

Loading

seiko2plus commented Apr 30, 2020 •

edited

Loading