[tex-live] bruno@clisp.org: relocatable packages

Karl Berry karl at freefriends.org
Sun Mar 30 13:56:35 CEST 2003


Bruno has just developed a general system for making packages
relocatable, see his detailed messages below.

Since we've expended a lot of effort on making the TeX system
relocatable (for TeX Live), I thought it would be worth forwarding.  It
would be nice if we could all use the same system at some point.

My only comment is that all this infrastructure is very nice, but it
would still not necessarily be obvious to me how to relocate a package,
because I don't know all the files that comprise it.

That is, say I installed a package in /usr/local on my machine (thus it
is intermixed with zillions of others), and then want to copy it to my
home directory on a shared machine.  If the package is relocatable, then
I can do this -- if I can find all the files.  But how do I know what
the files are?

For that matter, I've often wanted the file list for other reasons, too
-- for instance, to remove an old package.

Perhaps make install could write a manifest to $pkgdatadir/something?

K


Date: Sat, 29 Mar 2003 13:49:35 +0100 (CET)
From: Bruno Haible <bruno at clisp.org>
To: bug-gnulib at gnu.org
cc: Alexandre Duret-Lutz <duret_g at lrde.epita.fr>
cc: Akim Demaille <akim at epita.fr>
Subject: [Bug-gnulib] relocatable packages

Hi all,

It has been a pain for many users of GNU packages for a long time that
packages are not relocatable. It means a user cannot copy a program,
installed by another user on the same machine, to his home directory,
and have it work correctly (including i18n). So many users need to go
through "configure; make; make install" with all its dependencies,
options and hurdles.

rpms and similar package systems solve the "ease of installation"
problem but have a drawback: they hardwire path names, usually to /usr
or /usr/local, with the consequences that
  - Users need root privileges to install a binary package,
  - It is impossible to have two different versions of the same
    package installed (which makes Unix not better than Woe32 in this
    aspect).

So I've now made GNU gettext relocatable. Thanks to automake and
gnulib, it is foreseeable that other GNU packages, like GNU tar, GNU
coreutils, GNU libiconv etc., could support --enable-relocatable
without much maintainer effort. With some support in automake the
effort will be even less than it is now.

Here is a writeup:
  - how I came to the current implementation in
    http://www.haible.de/bruno/gnu/gettext-0.12-pre2.tar.gz
  - how the user/installer sees relocatability
  - what the maintainer has to do in order to provide --enable-relocatable.

All kinds of comments are welcome.


How to make GNU packages relocatable?
=====================================

Issues:
  I) explicit references to data file (in $pkgdatadir), to executables
     (in $pkglibdir) etc.
  II) shared libraries linked to executables or referenced by other shared
      libraries

Solutions:
  I) Pathname hacking based on one of
     1) A package dependent environment variable.
     2) Symlink from /var/installprefix/<package> to where the package is.
        (Not Woe32.)
     3) An explicit way to get the executable's pathname
        - Linux: readlink /proc/self/exe
        - Woe32: GetModuleFileName(NULL,...)
        - other Unixes (except BeOS): analyze argv[0], using PATH and
          realpath.
  II) Influence shared library runtime path:
      1) A package dependent environment variable (only Linux, Solaris, OSF/1).
      2) Symlink from /var/installprefix/<package> to where the package is.
         (Not Woe32.)
      4) Use a wrapper program around each program that sets the
         variable LD_LIBRARY_PATH (most Unixes, or LIBPATH on AIX, or
         SHLIB_PATH on HP-UX, or LIBRARY_PATH on BeOS, or
         DYLD_LIBRARY_PATH on MacOS X, or PATH on Woe32).
      5) Force sysadmin to modify /etc/ld.so.conf (not all Unixes).
      6) Put all shared libraries into $bindir. (Only Woe32.)

Evaluation/Drawbacks:
  1) usability problem
  2) Installation requires root privilege.
  3) ok
  4) expensive at runtime but easy to use
  5) Installation requires root privilege.
  6) ok

Optimal solution:
  Linux: I.3 + II.4
  Woe32: I.3 + II.6

Relocatability of libraries:
  Scenario:
    1) Install library L,
    2) Move it to a new prefix,
    3) Install package P that uses L,
    4) Move L and P to new locations.
  Does P still work? There are three cases:
    4a) Move P but keep L where it is.
    4b) Move P and L independently (assumes they were installed in different
        prefixes in step 3).
    4c) Move P and L together to a new location (assumes they were installed
        in the same prefix in step 3).
  Evaluation if L is a shared library:
    4b) will never work because P will not know where to find L - issue II -
        except if the user sets LD_LIBRARY_PATH or similar. Not important.
    4c) should work. Issue II is already done because is is the same as if
        step 2 had been omitted. But for issue I, we see that the data
        references of L and of P have to be relocated differently: For L,
        it must take into account the installation directory of step 1,
        while for P, it must take into account the installation directory of
        step 3. Still, the new location of P can be used to compute the new
        relocation rules for L.
    4a) should work. Issue II can be solved by hardwiring library pathnames,
        as is done usually with --enable-rpath. Issue I requires that data
        references of L and P are relocated differently, as for case 4c),
        but also that L must find its new location without referring to
        P's new location. Only works on Linux (via /proc/self/maps) and
        Woe32 (via GetModuleFileName in DllMain).
  Evaluation if L is a normal ar library:
    4a) will never work.
    4b) will never work.
    4c) should work.
  So we implement a solution for 4c) on all platforms and for 4a) on
  Linux, Woe32.

configure arguments:
  --enable-relocatable


User's, installer's view:
=========================

  --enable-relocatable makes the entire installed packages relocatable.
  This means, it can be moved or copied to a different location on the
  filesystem.
  It is possible to make symlinks to the installed and moved programs,
  and invoke them through the symlink. It is possible to do the same thing
  with a hard link _only_ if the hard linked file is in the same directory
  as the real program.

  For reliability it is best to give together with --enable-relocatable a
  --prefix option pointing to an otherwise unused (and never used again)
  directory, for example, --prefix=/tmp/inst$$. This is recommended
  because on some OSes the executables remember the location of shared
  libraries (and prefer them over LD_LIBRARY_PATH !), therefore such an
  executable will look for its shared libraries first in the original
  installation directory and only then in the current installation directory.

  Installation with --enable-relocatable will not work for setuid / setgid
  executables. (This is because such an executable kills its LD_LIBRARY_PATH
  variable when it is launched.)

  The runtime penalty and size penalty are nearly zero on Linux (just one
  system call more when an executable is launched), and small on other
  systems (the wrapper program just sets an environment variable and execs
  the real program).


Maintainer's view:
==================

  How to add relocatable support to a package?

  I assume you use automake. (Without automake the Makefile.in changes
  get so big that I'd recommend to switch to automake first.)

  I assume you have a lib/ directory in which you have auxiliary modules,
  from gnulib and others.

  * config.libpath
  * reloc-ldflags
  * install-reloc
    Add these scripts to your aux directory (AC_AUX_DIR argument), or to
    the top level directory if you have none.
  - config.libpath is used during autoconfiguration to determine the handling
    of shared library search paths by your system. It is a separate script so
    support for new systems can be added to it without the need to regenerate
    all configure files.
  - reloc-ldflags is invoked by the Makefiles to determine some additional
    flags to be used when linking, for systems supporting this method of
    relocatability.
  - install-reloc is invoked by the Makefiles to install a program in a
    relocatable way.

  * lib/ sources
    You need to add the following source files:

    + For use by your programs:

      - relocatable.h, relocatable.c
        Provides the function relocate() that transforms an installation-time
        pathname to a run-time pathname.

      - progname.h, progname.c
        Provides the function set_program_name() that you must call. It
        performs the preparations for relocate() to work later.

      - xreadlink.h, xreadlink.c
        Part of set_program_name()'s job is to determine the full pathname
        of the program at run time. On Linux, it does this by reading the
        symlink /proc/self/exe provided by the kernel. The xreadlink()
        function is a safe variant of readlink(), for this purpose.

      - canonicalize.h, canonicalize.c
        Part of set_program_name()'s job is to determine the full pathname
        of the program at run time. On other platforms than Linux, this is
        done by looking at argv[0]. But argv[0] can be a symlink, which must
        be resolved first. If we're unlucky enough that this symlink contains
        "../" components, simple string operations are not enough for
        resolving it; we need the full power of realpath() or
        canonicalize_file_name().

      - xmalloc.h, xmalloc.c, xstrdup.c
        Failsafe memory allocation. The modules above need it.

      - error.h, error.c
        Pretty error signalling routine. xmalloc needs it.

      - pathmax.h
        Usual pathname size, needed by canonicalize.c.

      - alloca_.h, alloca.c
        Memory allocation on the stack, needed by canonicalize.c.

      - memmove.c
        Needed by canonicalize.c.

      - stdbool.h.in
        Needed by progname.c.

    + For the wrapper programs:

      - relocwrapper.c
        This is the wrapper itself. Its purpose is to find out where
        the real program is, and invoke it.

      - setenv.h, setenv.c, unsetenv.c
        The wrapper program needs to set the LD_LIBRARY_PATH variable (or
        another, similar variable depending on the OS) to ensure that the
        real program will find its shared libraries after it has moved
        (together with its shared libraries).

      - strerror.c
        The wrapper program needs to print an error message when it cannot
        execute the real program. It needs strerror().

  * lib/Makefile.am

    Assuming that your Makefile gathers all modules in a libfoo.a, you'll
    have to add:
    - to libfoo_a_SOURCES
        error.h error.c
        pathmax.h
        progname.h progname.c
        xmalloc.h xmalloc.c xstrdup.c
        xreadlink.h xreadlink.c
    - to libfoo_a_LIBADD
        @ALLOCA@ @LIBOBJS@
        (or @LTALLOCA@ @LTLIBOBJS@ if you link libfoo with libtool)
    - to EXTRA_DIST
        alloca_.h alloca.c
        canonicalize.h canonicalize.c
        memmove.c
        relocatable.h relocatable.c
        relocwrapper.c
        setenv.h setenv.c unsetenv.c
        strerror.c
    - the rules for creating alloca.h if necessary
    - the rules for creating stdbool.h if necessary

  * m4/ macros

    Add these files:
      m4/alloca.m4
      m4/canonicalize.m4
      m4/error.m4
      m4/lib-ld.m4
      m4/onceonly.m4
      m4/pathmax.m4
      m4/relocatable.m4
      m4/setenv.m4
      m4/stdbool.m4
      m4/strerror.m4
      m4/xalloc.m4
      m4/xreadlink.m4

  * m4/Makefile.am

    Add the added files to EXTRA_DIST.

  * configure.ac

    Ensure it has the following invocations:
      AC_RELOCATABLE
      gl_FUNC_ALLOCA
      gl_CANONICALIZE
      gt_FUNC_ERROR_AT_LINE
      AC_REPLACE_FUNCS([memmove])
      gl_PATHMAX
      gt_FUNC_SETENV
      gt_STDBOOL_H
      gl_FUNC_STRERROR
      gl_XALLOC
      gl_XREADLINK

  * Your sources

    1) In every program, add to main() as the first statement (even before
       setting the locale or doing anthing related to libintl):

         set_program_name (argv[0]);

    2) Everywhere where you use a constant pathname from installation-time,
       wrap it in relocate() so it gets translated to the run-time situation.
       Example:

         bindtextdomain (PACKAGE, LOCALEDIR);

       becomes:

         bindtextdomain (PACKAGE, relocate (LOCALEDIR));

    3) In every shell script, add near the beginning the "Support for
       relocatability" section from gettextize.in. You can take the two
       functions without modifications; only the following need to be
       adapted:

       if test "@RELOCATABLE@" = yes; then
         exec_prefix="@exec_prefix@"
         bindir="@bindir@"
         orig_installdir="$bindir" # see Makefile.am's *_SCRIPTS variables
         func_find_curr_installdir # determine curr_installdir
         func_find_prefixes
         # Relocate the directory variables that we use.
         gettext_dir=`echo "$gettext_dir/" | sed -e "s%^${orig_installprefix}/%${curr_installprefix}/%" | sed -e 's,/$,,'`
       fi

       Here you adapt the definition of orig_installdir, depending on where
       the script gets installed. And at the end, instead of gettext_dir you
       transform those variables that you need.

  * Your Makefile.am

    For every program foo, that gets installed in, say, $(bindir), you add

      foo_CFLAGS = -DINSTALLDIR=\"$(bindir)\"
      if RELOCATABLE_VIA_LD
      foo_LDFLAGS = `$(RELOCATABLE_LDFLAGS) $(bindir)`
      endif

    And the following also also necessary, once per Makefile.am that installs
    a program:

      # Support for relocatability.
      RELOCATABLE_LIBRARY_PATH = $(libdir)
      RELOCATABLE_SRC_DIR = $(top_srcdir)/lib
      RELOCATABLE_BUILD_DIR = ../lib
      RELOCATABLE_CONFIG_H_DIR = ..
      @SET_RELOCATABLE@

    Here RELOCATABLE_LIBRARY_PATH denotes the directories in which your
    package (or any package you rely on, for example, gettext-runtime)
    has installed libraries. It is a colon separated list.

    RELOCATABLE_SRC_DIR denotes where to find relocwrapper.c and the related
    sources, so you need them only once in your package, even if you have
    multiple directories which install programs.

    RELOCATABLE_CONFIG_H_DIR denotes where to find the config.h which defines
    the various HAVE_* macros that relocwrapper.c and the related sources
    need.

    @SET_RELOCATABLE@ is some black magic that is filled in by the
    AC_RELOCATABLE macro.



_______________________________________________
Bug-gnulib mailing list
Bug-gnulib at gnu.org
http://mail.gnu.org/mailman/listinfo/bug-gnulib


More information about the tex-live mailing list