--as-needed introduction and fixing guide

1.  Introduction

What is --as-needed?

The --as-needed flag is passed to the GNU linker (GNU ld). The flag tells the linker to link in the produced binary only the libraries containing symbols actually used by the binary itself. This binary can be either a final executable or another library.

In theory, when linking something, only the needed libraries are passed to the command line used to invoke the linker. But to workaround systems with broken linkers or not using ELF format, many libraries declare some "dependencies" that get pulled in while linking. A simple example can be found by looking at the libraries declared as dependencies by gtk+ 2.0:

$ pkg-config gtk+-2.0 --libs
-lgtk-x11-2.0 -lgdk-x11-2.0 -latk-1.0 -lgdk_pixbuf-2.0 -lm -lpangocairo-1.0 -lpango-1.0 -lcairo -lgobject-2.0 -lgmodule-2.0 -ldl -lglib-2.0

Of course, if the application is just using functions from gtk+ 2.0, a simple link line with -lgtk-x11-2.0 should make it build fine, but looking at which libraries are needed and which are not from a package point of view is often an impossible task.

How can --as-needed be useful?

The use of the --as-needed flag allows the linker to avoid linking extra libraries in a binary. This not only improves startup times (as the loader does not have to load all the libraries for every step) but might avoid the full initialization of things like KDE's KIO for a binary if it's not using the KIO framework.

More importantly, the use of --as-needed avoids adding dependencies to a binary that are prerequisites of one of its direct or indirect dependencies. This is important because when a library changes SONAME after an ABI change, all the binaries directly linking to it have to be rebuilt. By linking only the libraries that are actually needed, the breakage due to an ABI change is reduced. It is particularly useful when the ABI breakage happens in a library used by some other high level library (like cairo, which is used directly by gtk+-2.0, and gets linked indirectly in applications using the latter), as it prevents the rebuild of the final binaries and thus of the packages carrying them.

It is also useful to check whether the dependencies stated by the documentation are actually used by a package: it's not impossible that a package checks in a configure script for some library, and then links to it, but without using it at all because the code using it was removed or refactored or has not been written.

How to use --as-needed

If you want to try using the --as-needed flag, you can simply add it to your /etc/make.conf file. Note that LDFLAGS are generally passed not directly to ld but to gcc, so you have to use the -Wl, prefix to pass them back to the linker.

LDFLAGS="-Wl,--as-needed"

This flag should (note again the conditional) work on every Linux platform supported by binutils, but tests are being done only on AMD64 at the moment (some unofficial testing is being done on PPC too, but problems with binutils mix up their validity). It's known not to work on FreeBSD and probably does not work on other non-Linux targets.

As --as-needed depends not only on the way the package you're building is linked but also its dependencies, there are quite a few packages that were silently patched and fixed and might require rebuilding. Please make sure to rebuild the dependencies failing to link against before filing a bug.

LDFLAGS="-Wl,--hash-style=gnu,-O1 -Wl,--as-needed"

Forced --as-needed

Using --as-needed through LDFLAGS is the suggested method for standard users and users who don't want to have too many problems with packages failing to build. For developers, power users, and people who want to stress test buildsystems, a different strategy is also available, that forces each and every build to use --as-needed during the linking phase, as long as the gcc frontend is used as linker.

The forced approach is useful to test packages that don't respect the LDFLAGS variable or packages that incorrectly filter the flag (see the next section for details about that), and to work around the libtool reordering bug. A compiler set to force --as-needed is easily reversed so that if a package is needed to be built without the flag a single switch is required.

To force the --as-needed flag on the compiler, GCC spec files are used, creating a new profile for the compiler that always use it during linking phase. For any version of gcc it is possible to build the modified spec file using the following series of commands

# export SPECSFILE=$(dirname "$(gcc -print-libgcc-file-name)")/asneeded.specs
# export CURRPROFILE=/etc/env.d/gcc/$(gcc-config -c)
# gcc -dumpspecs | sed -e '/link:/,+1 s:--eh-frame-hdr:\0 --as-needed:' > "$SPECSFILE"
# sed "${CURRPROFILE}" -e '1i\GCC_SPECS='$SPECSFILE > "${CURRPROFILE}-asneeded"
# gcc-config "$(basename "${CURRPROFILE}")-asneeded"
# source /etc/profile

To switch between the --as-needed and standard compilers, just use gcc-config like they were different compiler versions or hardened and standard compilers.

Properly filtering --as-needed

Sometimes it is needed to filter --as-needed as the software is designed in such a way that it is not fixable to make use of it. Unfortunately just filtering the flag is often not the best choice because LDFLAGS can be tricky (they can be concatenated through commas, making it impossible to filter them out), and because it is possible to set it as a default through GCC's spec files.

If you really need to disable the --as-needed behaviour, because of design choices that conflicts with the way --as-needed works (and thus not just because the package is broken, and fixable, when building with --as-needed), what you should be doing is something along these lines.

inherit flag-o-matic
...

pkg_setup() {
    append-ldflags $(no-as-needed)
}

2.  Fixing problems with --as-needed

Identifying the problem

Developers willing to fix failures related to the --as-needed flag should be aware that there are many cases of failures that may fall into one of a few different categories. I'll try to explain here the reasoning behind the failures and ways to fix them; some of them are really simple, others are not.

If you find a package failing when using --as-needed you may file a bug in Gentoo Bugzilla, blocking bug #129413.

Failure in compile, unrecognized option

This is at the same time the most simple and the most annoying problem that can be found. There might be packages failing with an error like the following after adding -Wl,--as-needed to LDFLAGS:

ld: unrecognized option '-Wl,--as-needed'
ld: use the --help option for usage information

This is caused by ld being called with the LDFLAGS variable instead of gcc, thus breaking because it doesn't recognize the -Wl, prefix used to tell gcc to pass the option down to the linker. To fix this, one must pass the output of the raw-ldflags function to the make process.

inherit flag-o-matic
...

src_compile() {
    emake LDFLAGS="$(raw-ldflags)" || die "emake failed"
}

Failure in final linking, undefined symbols

This is the most common error that happens while using --as-needed. It happens during the final linking stage of an executable (libraries don't create problems, because they are allowed to have undefined symbols). The executable linking stage dies because of an undefined symbol that is present in one of the libraries fed to the command line. However, the library is not used by the executable itself, thus it gets removed by --as-needed.

This usually means that a library was not linked to another library, but was using it, and then relying on the final executable to link them together. This behavior is also an extra encumbrance on developers using that library because they have to check for the requirements.

The fix to this kind of problem is usually simple: just find which library provides the symbols and which one is requiring them (the error message from the linker should contain the name of the latter). Then make sure that when the library is linked from the source files it's also linked to the first. While using autotools, the dependent library has to be checked in the configure (this should already be the case to specify the dependencies in the pkg-config data file or in the script provided) and then the variable carrying this value should be added to the LIBADD variable for the library to be built.

Failure in execution, undefined symbols

Sometimes the undefined symbol errors don't happen while linking, but rather at the execution of an application built with --as-needed. The cause, however, is just the same as for the undefined symbols in linking: a directly-linked library did not link one of its dependencies. It also has the same solution: find which library carries the undefined symbols and make sure that it gets linked to the library providing them.

Failure in ./configure

Albeit less common than other kind of failures, ./configure execution can fail because of --as-needed too. With this kind of failures, though, it's difficult to give a single and simple solution, as there are multiple reasons that might make the script fail.

The most common option between those that we have now is that a library is checked for, but that library wasn't linked to all its prerequisites (and thus required them to be passed afterward). As --as-needed makes the linker ignore all the libraries not needed by the current target, the prerequisites will result missing.

To check a configure.ac or configure.in file for this kind of failures, you can look for the AC_CHECK_LIB macro, and see whether the fifth parameter is used (other-libraries). When it is, it often means that those libraries need to be linked in the final binary to satisfy the dependencies of the library to check for. At that point, the library need to be fixed.

AC_CHECK_LIB([foo], [foo_something],
             [have_foo=yes], [have_foo=no],
             [-ldl -lnsl])

Another possible failure during ./configure execution happens when the code is going to check for functions or other particular items (symbols, behaviour) and mistakenly pass the dependency libraries through the LDFLAGS variable.

PKG_CHECK_MODULES([FOO], [foo])

...

save_LDFLAGS=$LDFLAGS
LDFLAGS="$LDFLAGS $FOO_LIBS"

AC_CHECK_FUNCS([foo_choice1 foo_choice2 foo_choice3])

LDFLAGS=$save_LDFLAGS

In this case the solution is very quick: just replace LDFLAGS with LIBS in both saving and restoring. This way the libraries will be passed in the correct order (see the following section for more insight about this kind of problem).

Importance of linking order

One thing that might be frustrating when fixing packages for --as-needed is that sometimes, although all the libraries are present in the linking line, they are just ignored and not linked at all. This leads to the same issues as above; missing symbols either during final link or execution. This is because of a behaviour of the GNU linker that's enforced by --as-needed.

Basically, what the linker does is look for the symbols missing in a given file (either an object file, a static archive or a library) only in the files coming after it. When using the normal linking, without --as-needed, this is not a problem, although there might be some internal drawbacks on the linking stage, the files are linked together without considering ordering. But with the flag, the libraries that aren't used for resolving symbols are discarded and thus not linked.

(in this case libm is considered before the object files and discarded independently from the content of the two.)
$ gcc -Wl,--as-needed -lm someunit1.o someunit2.o -o program

(this is the correct linking order to get libm linked only if needed.)
$ gcc -Wl,--as-needed someunit1.o someunit2.o -lm -o program

The fix in this case is to simply fix the linking order so that the libraries given to the linker are all after the object files and the static archives.

While using autotools there are usually small cases where this happens, because usually libs are fed either via the LIBS variable in the configure script or are listed in the LDADD/LIBADD variables for the target which is being built. The only case when this happens to be a problem is when the libraries get fed into LDFLAGS variable (which is incorrect).

Initializers and Deconstructors

There exists a class of applications at the moment that break when using --as-needed. These applications are not at fault, but rather the linker itself. The linker is unable to detect dependencies between the initializers and deconstructors (.init/.fini ELF sections) when working with C++ code. As such, it may discard libraries when none of the symbols are used from it, thus mistakenly changing the initialization and deconstruction code paths.

While this class of applications is small and there are no known applications yet which fall into this category, this is something to keep in mind. The only way to really detect such a thing is by proper source code and runtime analysis.

The contents of this document, unless otherwise expressly stated, are licensed under the CC-BY-SA-2.5 license. The Gentoo Name and Logo Usage Guidelines apply.