--as-needed introduction and fixing guide
1.
Introduction
What is --as-needed?
Note:
This guide is related to an experimental feature or function that may or
may not work on your system, packages, or architecture. The information stated
here is presented only for informative purposes, and is not officially supported.
Working patches to fix packages to work with this feature are usually welcome
if they don't create more problems for current users. In any case, please don't
report problems caused by the steps explained in this guide on bugzilla.
|
The --as-needed flag is passed to the GNU linker (GNU
ld). The flag tells the linker to link in the produced binary only the
libraries containing symbols actually used by the binary itself. This binary
can be either a final executable or another library.
In theory, when linking something, only the needed libraries are passed to the
command line used to invoke the linker. But to workaround systems with broken
linkers or not using ELF format, many libraries declare some "dependencies"
that get pulled in while linking. A simple example can be found by looking at
the libraries declared as dependencies by gtk+ 2.0:
Code Listing 1.1: libraries needed to link to gtk+ 2.0 |
$ pkg-config gtk+-2.0 --libs
-lgtk-x11-2.0 -lgdk-x11-2.0 -latk-1.0 -lgdk_pixbuf-2.0 -lm -lpangocairo-1.0 -lpango-1.0 -lcairo -lgobject-2.0 -lgmodule-2.0 -ldl -lglib-2.0
|
Of course, if the application is just using functions from gtk+ 2.0, a simple
link line with -lgtk-x11-2.0 should make it build fine, but looking at
which libraries are needed and which are not from a package point of view is
often an impossible task.
How can --as-needed be useful?
The use of the --as-needed flag allows the linker to avoid linking extra
libraries in a binary. This not only improves startup times (as the loader does
not have to load all the libraries for every step) but might avoid the full
initialization of things like KDE's KIO for a binary if it's not using the KIO
framework.
More importantly, the use of --as-needed avoids adding dependencies to a
binary that are prerequisites of one of its direct or indirect dependencies.
This is important because when a library changes SONAME after an ABI change,
all the binaries directly linking to it have to be rebuilt. By linking only the
libraries that are actually needed, the breakage due to an ABI change is
reduced. It is particularly useful when the ABI breakage happens in a library
used by some other high level library (like cairo, which is used
directly by gtk+-2.0, and gets linked indirectly in applications
using the latter), as it prevents the rebuild of the final binaries and thus of
the packages carrying them.
It is also useful to check whether the dependencies stated by the documentation
are actually used by a package: it's not impossible that a package checks in
a configure script for some library, and then links to it, but without using it
at all because the code using it was removed or refactored or has not been
written.
How to use --as-needed
Warning:
At the time of writing, there are many packages failing in funny ways because
of --as-needed as they weren't designed to be used with it. While there
shouldn't be (note the conditional) problems such as crashes, this flag
is not considered safe for production use and not supported in any way by
Gentoo.
|
If you want to try using the --as-needed flag, you can simply add it to
your /etc/make.conf file. Note that LDFLAGS are generally passed
not directly to ld but to gcc, so you have to use the -Wl,
prefix to pass them back to the linker.
Code Listing 1.2: enabling --as-needed in make.conf |
LDFLAGS="-Wl,--as-needed"
|
Note:
There are known issues with ld shipped with binutils series 2.16, and
early 2.17 prereleases. If you feel like trying this flag, you have to use at
least version 2.17, that has all the fixes needed to have --as-needed
behave correctly. Please note that 2.17.50.0.2 version is not working as
one might expect and is, in regard to --as-needed, older than 2.17, as it
misses a few important fixes.
|
This flag should (note again the conditional) work on every Linux
platform supported by binutils, but tests are being done only on AMD64 at the
moment (some unofficial testing is being done on PPC too, but problems with
binutils mix up their validity). It's known not to work on FreeBSD and probably
does not work on other non-Linux targets.
As --as-needed depends not only on the way the package you're building is
linked but also its dependencies, there are quite a few packages that were
silently patched and fixed and might require rebuilding. Please make sure to
rebuild the dependencies failing to link against before filing a bug.
2.
Fixing problems with --as-needed
Identifying the problem
Anyone willing to try out the --as-needed flag should be aware that
there are many cases of failures that may fall into one of a few different
categories. I'll try to explain here the reasoning behind the failures and
ways to fix them; some of them are really simple, others are not.
Once again, if you're not interested in fixing those problems, or you are not a
developer, please try to stay away from experimental things like this. If you're
a developer or you have a patch to fix --as-needed issues, you can file a bug in
Gentoo Bugzilla, blocking
bug #129413.
Binutils bugs
As stated above, the 2.16 binutils series and the early 2.17 prereleases suffer
from problems when linking with --as-needed, which leads to ld
segfaulting when trying to link complex C++ libraries like wxGTK.
In this case the first thing to do is to use the latest prerelease available
from upstream. At the date of writing the last prerelease is 2.16.92 that is
said to solve most if not all the problems with --as-needed flag and
does not fail anymore with wxGTK for once.
If the last snapshot doesn't work, you might consider filtering the flag with
the following code, after reporting the case upstream with as many details
possible.
Code Listing 2.1: filtering the --as-needed flag in an ebuild |
inherit flag-o-matic
...
pkg_setup() {
filter-ldflags -Wl,--as-needed --as-needed
}
|
Important:
These problems should be gone as of 2.16.92 and thus the filtering is
unnecessary, as anyone using --as-needed and finding segfaults should
simply update binutils. For this reason you might want to do this locally but
you should not commit flags filtering for this reason.
|
Failure in compile, unrecognized option
This is at the same time the most simple and the most annoying problem that can
be found. There might be packages failing with an error like the following
after adding -Wl,--as-needed to LDFLAGS:
Code Listing 2.2: common error while using --as-needed |
ld: unrecognized option '-Wl,--as-needed'
ld: use the --help option for usage information
|
This is caused by ld being called with the LDFLAGS variable
instead of gcc, thus breaking because it doesn't recognize the
-Wl, prefix used to tell gcc to pass the option down to the
linker. To fix this, one must pass the output of the raw-ldflags
function to the make process.
Code Listing 2.3: example of raw-ldflags usage |
inherit flag-o-matic
...
src_compile() {
emake LDFLAGS="$(raw-ldflags)" || die "emake failed"
}
|
Failure in final linking, undefined symbols
This is the most common error that happens while using --as-needed. It
happens during the final linking stage of an executable (libraries don't create
problems, because they are allowed to have undefined symbols). The executable
linking stage dies because of an undefined symbol that is present in one of
the libraries fed to the command line. However, the library is not used by the
executable itself, thus it gets removed by --as-needed.
This usually means that a library was not linked to another library, but was
using it, and then relying on the final executable to link them together. This
behavior is also an extra encumbrance on developers using that library because
they have to check for the requirements.
The fix to this kind of problem is usually simple: just find which library
provides the symbols and which one is requiring them (the error message from
the linker should contain the name of the latter). Then make sure that when the
library is linked from the source files it's also linked to the first. While
using autotools, the dependent library has to be checked in the configure (this
should already be the case to specify the dependencies in the pkg-config
data file or in the script provided) and then the variable carrying this value
should be added to the LIBADD variable for the library to be built.
Failure in execution, undefined symbols
Sometimes the undefined symbol errors don't happen while linking, but rather at
the execution of an application built with --as-needed. The cause, however, is
just the same as for the undefined symbols in linking: a directly-linked library
did not link one of its dependencies. It also has the same solution: find which
library carries the undefined symbols and make sure that it gets linked to the
library providing them.
Failure in ./configure
This is hopefully the less frequent failure but it's also one of the most
impractical to find, although is no different to fix than other failures.
Basically the ./configure script of an autotooled package can fail to
find a given library (and thus abort if the support is a strict requirement, or
simply ignore the presence of a library) if that library does not link against
its right dependencies.
The quickest way to find possible failures because of the above is to look into
AC_CHECK_LIB calls inside the configure script; when the fifth
parameter is used (other-libraries) it often means that those libraries
need to be linked in the same binary with the one to check, as they are not
linked in automatically. While this works in a normal situation, when using the
--as-needed flag, they are probably dropped because they are not used by
the test program.
The fixes are the usual ones explained above, but instead of being applied to
the failing package they have to be applied to the library that's failing to be
detected.
Note:
While applying patches to libraries to fix --as-needed support, it is
usually not the case to apply a revision bump: those who don't want to use the
flag don't need to rebuild the library. For this reason, those who want to use
--as-needed are invited to run an emerge -e world so that
libraries are rebuilt.
|
Importance of linking order
One thing that might be frustrating when fixing packages for --as-needed
is that sometimes, although all the libraries are present in the linking line,
they are just ignored and not linked at all. This leads to the same issues as
above; missing symbols either during final link or execution. This is because
of a behaviour of the GNU linker that's enforced by --as-needed.
Basically, what the linker does is look for the symbols missing in a given file
(either an object file, a static archive or a library) only in the files coming
after it. When using the normal linking, without --as-needed, this is
not a problem, although there might be some internal drawbacks on the linking
stage, the files are linked together without considering ordering. But with
the flag, the libraries that aren't used for resolving symbols are discarded
and thus not linked.
Code Listing 2.4: example of wrong and correct linking order |
$ gcc -Wl,--as-needed -lm someunit1.o someunit2.o -o program
$ gcc -Wl,--as-needed someunit1.o someunit2.o -lm -o program
|
The fix in this case is to simply fix the linking order so that the libraries given
to the linker are all after the object files and the static archives.
While using autotools there are usually small cases where this happens, because
usually libs are fed either via the LIBS variable in the
configure script or are listed in the LDADD/LIBADD variables for the target
which is being built. The only case when this happens to be a problem is when
the libraries get fed into LDFLAGS variable (which is incorrect).
Initializers and Deconstructors
There exists a class of applications at the moment that break when using
-Wl,--as-needed. These applications are not at fault, but rather the linker
itself. The linker is unable to detect dependencies between the initializers
and deconstructors (.init/.fini ELF sections) when working with C++ code. As
such, it may discard libraries when none of the symbols are used from it, thus
mistakenly changing the initialization and deconstruction code paths.
While this class of applications is small and there are no known applications
yet which fall into this category, this is something to keep in mind. The only
way to really detect such a thing is by proper source code and runtime
analysis.
The contents of this document are licensed under the Creative Commons -
Attribution / Share Alike license.
|