--as-needed introduction and fixing guide

1. Introduction

Note: This guide is related to an experimental feature or function that may or may not work on your system, packages, or architecture. The information stated here is presented only for informative purposes, and is not officially supported. Working patches to fix packages to work with this feature are usually welcome if they don't create more problems for current users. In any case, please don't report problems caused by the steps explained in this guide on bugzilla.

The --as-needed flag is passed to the GNU linker (GNU ld). The flag tells the linker to link in the produced binary only the libraries containing symbols actually used by the binary itself. This binary can be either a final executable or another library.

In theory, when linking something, only the needed libraries are passed to the command line used to invoke the linker. But to workaround systems with broken linkers or not using ELF format, many libraries declare some "dependencies" that get pulled in while linking. A simple example can be found by looking at the libraries declared as dependencies by gtk+ 2.0:

Code Listing 1.1: libraries needed to link to gtk+ 2.0

$ pkg-config gtk+-2.0 --libs
-lgtk-x11-2.0 -lgdk-x11-2.0 -latk-1.0 -lgdk_pixbuf-2.0 -lm -lpangocairo-1.0 -lpango-1.0 -lcairo -lgobject-2.0 -lgmodule-2.0 -ldl -lglib-2.0

Of course, if the application is just using functions from gtk+ 2.0, a simple link line with -lgtk-x11-2.0 should make it build fine, but looking at which libraries are needed and which are not from a package point of view is often an impossible task.

How can --as-needed be useful?

The use of the --as-needed flag allows the linker to avoid linking extra libraries in a binary. This not only improves startup times (as the loader does not have to load all the libraries for every step) but might avoid the full initialization of things like KDE's KIO for a binary if it's not using the KIO framework.

More importantly, the use of --as-needed avoids adding dependencies to a binary that are prerequisites of one of its direct or indirect dependencies. This is important because when a library changes SONAME after an ABI change, all the binaries directly linking to it have to be rebuilt. By linking only the libraries that are actually needed, the breakage due to an ABI change is reduced. It is particularly useful when the ABI breakage happens in a library used by some other high level library (like cairo, which is used directly by gtk+-2.0, and gets linked indirectly in applications using the latter), as it prevents the rebuild of the final binaries and thus of the packages carrying them.

It is also useful to check whether the dependencies stated by the documentation are actually used by a package: it's not impossible that a package checks in a configure script for some library, and then links to it, but without using it at all because the code using it was removed or refactored or has not been written.

How to use --as-needed

Warning: At the time of writing, there are many packages failing in funny ways because of --as-needed as they weren't designed to be used with it. While there shouldn't be (note the conditional) problems such as crashes, this flag is not considered safe for production use and not supported in any way by Gentoo.

If you want to try using the --as-needed flag, you can simply add it to your /etc/make.conf file. Note that LDFLAGS are generally passed not directly to ld but to gcc, so you have to use the -Wl, prefix to pass them back to the linker.

Code Listing 1.2: enabling --as-needed in make.conf

LDFLAGS="-Wl,--as-needed"

Note: There are known issues with ld shipped with binutils series 2.16, and early 2.17 prereleases. If you feel like trying this flag, you have to use at least version 2.17, that has all the fixes needed to have --as-needed behave correctly. Please note that 2.17.50.0.2 version is not working as one might expect and is, in regard to --as-needed, older than 2.17, as it misses a few important fixes.

This flag should (note again the conditional) work on every Linux platform supported by binutils, but tests are being done only on AMD64 at the moment (some unofficial testing is being done on PPC too, but problems with binutils mix up their validity). It's known not to work on FreeBSD and probably does not work on other non-Linux targets.

As --as-needed depends not only on the way the package you're building is linked but also its dependencies, there are quite a few packages that were silently patched and fixed and might require rebuilding. Please make sure to rebuild the dependencies failing to link against before filing a bug.

2. Fixing problems with --as-needed

Identifying the problem

Anyone willing to try out the --as-needed flag should be aware that there are many cases of failures that may fall into one of a few different categories. I'll try to explain here the reasoning behind the failures and ways to fix them; some of them are really simple, others are not.

Once again, if you're not interested in fixing those problems, or you are not a developer, please try to stay away from experimental things like this. If you're a developer or you have a patch to fix --as-needed issues, you can file a bug in Gentoo Bugzilla, blocking bug #129413.

Binutils bugs

As stated above, the 2.16 binutils series and the early 2.17 prereleases suffer from problems when linking with --as-needed, which leads to ld segfaulting when trying to link complex C++ libraries like wxGTK.

In this case the first thing to do is to use the latest prerelease available from upstream. At the date of writing the last prerelease is 2.16.92 that is said to solve most if not all the problems with --as-needed flag and does not fail anymore with wxGTK for once.

If the last snapshot doesn't work, you might consider filtering the flag with the following code, after reporting the case upstream with as many details possible.

Code Listing 2.1: filtering the --as-needed flag in an ebuild

inherit flag-o-matic
...

pkg_setup() {
    filter-ldflags -Wl,--as-needed --as-needed
}

Important: These problems should be gone as of 2.16.92 and thus the filtering is unnecessary, as anyone using --as-needed and finding segfaults should simply update binutils. For this reason you might want to do this locally but you should not commit flags filtering for this reason.

Failure in compile, unrecognized option

This is at the same time the most simple and the most annoying problem that can be found. There might be packages failing with an error like the following after adding -Wl,--as-needed to LDFLAGS:

Code Listing 2.2: common error while using --as-needed

ld: unrecognized option '-Wl,--as-needed'
ld: use the --help option for usage information

This is caused by ld being called with the LDFLAGS variable instead of gcc, thus breaking because it doesn't recognize the -Wl, prefix used to tell gcc to pass the option down to the linker. To fix this, one must pass the output of the raw-ldflags function to the make process.

Code Listing 2.3: example of raw-ldflags usage

inherit flag-o-matic
...

src_compile() {
    emake LDFLAGS="$(raw-ldflags)" || die "emake failed"
}

Failure in final linking, undefined symbols

This is the most common error that happens while using --as-needed. It happens during the final linking stage of an executable (libraries don't create problems, because they are allowed to have undefined symbols). The executable linking stage dies because of an undefined symbol that is present in one of the libraries fed to the command line. However, the library is not used by the executable itself, thus it gets removed by --as-needed.

This usually means that a library was not linked to another library, but was using it, and then relying on the final executable to link them together. This behavior is also an extra encumbrance on developers using that library because they have to check for the requirements.

The fix to this kind of problem is usually simple: just find which library provides the symbols and which one is requiring them (the error message from the linker should contain the name of the latter). Then make sure that when the library is linked from the source files it's also linked to the first. While using autotools, the dependent library has to be checked in the configure (this should already be the case to specify the dependencies in the pkg-config data file or in the script provided) and then the variable carrying this value should be added to the LIBADD variable for the library to be built.

Failure in execution, undefined symbols

Sometimes the undefined symbol errors don't happen while linking, but rather at the execution of an application built with --as-needed. The cause, however, is just the same as for the undefined symbols in linking: a directly-linked library did not link one of its dependencies. It also has the same solution: find which library carries the undefined symbols and make sure that it gets linked to the library providing them.

Failure in ./configure

This is hopefully the less frequent failure but it's also one of the most impractical to find, although is no different to fix than other failures. Basically the ./configure script of an autotooled package can fail to find a given library (and thus abort if the support is a strict requirement, or simply ignore the presence of a library) if that library does not link against its right dependencies.

The quickest way to find possible failures because of the above is to look into AC_CHECK_LIB calls inside the configure script; when the fifth parameter is used (other-libraries) it often means that those libraries need to be linked in the same binary with the one to check, as they are not linked in automatically. While this works in a normal situation, when using the --as-needed flag, they are probably dropped because they are not used by the test program.

The fixes are the usual ones explained above, but instead of being applied to the failing package they have to be applied to the library that's failing to be detected.

Note: While applying patches to libraries to fix --as-needed support, it is usually not the case to apply a revision bump: those who don't want to use the flag don't need to rebuild the library. For this reason, those who want to use --as-needed are invited to run an emerge -e world so that libraries are rebuilt.

Importance of linking order

One thing that might be frustrating when fixing packages for --as-needed is that sometimes, although all the libraries are present in the linking line, they are just ignored and not linked at all. This leads to the same issues as above; missing symbols either during final link or execution. This is because of a behaviour of the GNU linker that's enforced by --as-needed.

Basically, what the linker does is look for the symbols missing in a given file (either an object file, a static archive or a library) only in the files coming after it. When using the normal linking, without --as-needed, this is not a problem, although there might be some internal drawbacks on the linking stage, the files are linked together without considering ordering. But with the flag, the libraries that aren't used for resolving symbols are discarded and thus not linked.

Code Listing 2.4: example of wrong and correct linking order

(in this case libm is considered before the object files and discarded independently from the content of the two.)
$ gcc -Wl,--as-needed -lm someunit1.o someunit2.o -o program

(this is the correct linking order to get libm linked only if needed.)
$ gcc -Wl,--as-needed someunit1.o someunit2.o -lm -o program

The fix in this case is to simply fix the linking order so that the libraries given to the linker are all after the object files and the static archives.

While using autotools there are usually small cases where this happens, because usually libs are fed either via the LIBS variable in the configure script or are listed in the LDADD/LIBADD variables for the target which is being built. The only case when this happens to be a problem is when the libraries get fed into LDFLAGS variable (which is incorrect).

Initializers and Deconstructors

There exists a class of applications at the moment that break when using -Wl,--as-needed. These applications are not at fault, but rather the linker itself. The linker is unable to detect dependencies between the initializers and deconstructors (.init/.fini ELF sections) when working with C++ code. As such, it may discard libraries when none of the symbols are used from it, thus mistakenly changing the initialization and deconstruction code paths.

While this class of applications is small and there are no known applications yet which fall into this category, this is something to keep in mind. The only way to really detect such a thing is by proper source code and runtime analysis.

The contents of this document are licensed under the Creative Commons - Attribution / Share Alike license.

Updated April 17, 2007

Summary: This guide is meant to explain how the --as-needed LDFLAG works and to provide steps to fix simple cases where --as-needed fails to build a package.

Diego Pettenò
Author

Donate to support our development efforts.