Debugging Loading Issues

Posted on

Sometimes an application works in your developer environment but breaks elsewhere, usually on startup. You may get an error message like:

<app>: error while loading shared libraries: lib<libname>.so: cannot open shared object file: No such file or directory

These are dynamic linker (loader) issues, and this page is a cheat-sheet to debugging them.

https://developer.ibm.com/tutorials/l-dynamic-libraries

The Linker and Dynamic Linker

The last stage of building an application is to link it. The command line options related to linking contain two relevant details: the -l<libname> option to specify the libraries to link to, and the -L<directory> to specify additional directories to search for those libraries. As the application is linked, all symbols must be resolved via the libraries added via -l, which are found in some default paths or the paths specified via -L. If any symbol is missing, there will be a link-time error that looks like this:

/usr/bin/ld: <file.o>: in function <function>': file.c:(.text+0x__): undefined reference to <missing function>'

Those issues are discovered when the program is linked. However, once those issues are fixed and the program finally links, there is still no guarantee that it will run.

When a program is launched, the kernel runs the program through the dynamic linker, also called the loader (man ld.so). The loader will be something like /lib64/ld-linux-x86-64.so.2. This is different from the linker ld (usually /usr/bin/ld – see man ld) which is used to link the application in the final stage of building.

The linker can be told where to look for libraries using the -L flag, whereas the dynamic linker/loader has its own algorithm to find libraries. A simplified list of directories it looks in (in-order) are:

  1. [[deprecated]] From the applications hardcoded DT_RPATH if set (usually via passing -rpath to an older linker), and if DT_RUNPATH (in step 3) is not set
  2. Looks in directories from the LD_LIBRARY_PATH environment variable
  3. From the applications hardcoded DT_RUNPATH, if set (via passing -rpath to the linker)
  4. Directories listed in a cache file /etc/ld.so.cache
    • The cache can be updated via running ldconfig (man ldconfig) with a directory to search for libraries as the argument (or set in LD_LIBRARY_PATH).
    • The cache contains lists of libraries found in directories specified by /etc/ld.so.conf and /etc/ld.so.conf.d/*.conf
  5. /lib, or /lib64
  6. /usr/lib, or /usr/lib64

A common mistake is to build a new library, move it to something like /usr/local/lib on a new system, but forget to update the cache to tell it there’s a new library there. Then, running the application will result in the dynamic linker error mentioned before (error while loading shared libraries). To update the cache, simply run sudo ldconfig, or sudo ldconfig <path/to/library> if the path/to/library is not in one of the /etc/ld.so.conf.d/* files.

Below are a number of ways to debug linking issues for an application named app.

LD_DEBUG

The first option is to set the LD_DEBUG environment variable. I find the most useful setting to be libs, but there are other options. Set it to help to get more information, or read the ld.so man page.

Bash
$ LD_DEBUG=libs app
9724: find library=libmylib.so [0]; searching
9724: search path=[...]:build/lib (RUNPATH from file app)
9724: trying file=build/lib/glibc-hwcaps/x86-64-v3/libmylib.so
[...]
9724: search cache=/etc/ld.so.cache
9724: search path=/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v3:[...]:/usr/lib (system search path)
[...]
9724: trying file=/lib/libmylib.so
[...]
9724: trying file=/usr/lib/libmylib.so
9724:
app: error while loading shared libraries: libmylib.so: cannot open shared object file: No such file or directory

LD_LIBRARY_PATH

If a library is not found but you know the exact path to it, you can set the LD_LIBRARY_PATH environment variable. This will inform the linker to look in that specific directory when searching for libraries to load.

Bash
$ LD_LIBRARY_PATH=/path/to/library app

If this works, then the path to the library should be stored in a global setting somewhere: perhaps in /etc/ld.so.conf or one of the /etc/ld.so.conf.d/*.conf files. You can also update the library cache by running ldconfig /path/to/library as sudo.

file

file is a tool that can give basic information about a file: what type of a file is it? Is it a text file, or is it meant to be executed? Is it statically or dynamically linked? How is it run?

Examples:

Bash
$ file app
app: ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=b1e59f71d5d62fd7d3f95bde40cea90d39bd9053, for GNU/Linux 3.2.0, not stripped
Bash
$ file app
app: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=fec30f5745c2582aaf143e7079d3bb46ca7a010f, for GNU/Linux 3.7.0, stripped

objdump

Bash
$ objdump -f app
app:     file format elf64-x86-64
architecture: i386:x86-64, flags 0x00000150:
HAS_SYMS, DYNAMIC, D_PAGED
start address 0x000000000000e780

readelf

Bash
$ readelf -h app
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 03 00 00 00 00 00 00 00 00 
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - GNU
  ABI Version:                       0
  Type:                              DYN (Position-Independent Executable file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0xe780
  Start of program headers:          64 (bytes into file)
  Start of section headers:          1794376 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         13
  Size of section headers:           64 (bytes)
  Number of section headers:         41
  Section header string table index: 40

ldd

Bash
$ ldd app
        linux-vdso.so.1 (0x00007ffdeb55d000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fe32dee1000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fe32ddfa000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fe32ddda000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe32dbb1000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fe32e1b9000)

nm

Dump all symbols in a binary (recommend to grep to search for something):

$ nm -C app

patchelf

patchelf is a utility to patch an already-built binary so that it can:

  1. Set an r-path (that would have been set by the ld linker using the -rpath option)
  2. Remove/add/replace needed libraries from a binary
  3. Set the name of a library

More information can be found on its github page: https://github.com/NixOS/patchelf

It can be built from source or installed through pip: pip install patchelf