Library versioning

From ParabolaWiki
Jump to: navigation, search


1 Introduction

When building a package with makepkg or libremakepkg, most of the time the binaries (if any) in the package being built are linked against library that are in other packages.

If the library it depends on is missing or change soname (libsomething.so.1.23) in a way that is incompatible, the package don't work anymore.

For instance if Arch Linux 32 update libdvdread package and that libdvdread.so.7 is now libdvdread.so.8 in the new package, then software like mplayer will stop working, and will output an error like that:

$ mplayer
mplayer: error while loading shared libraries: libdvdread.so.7: cannot open shared object file: No such file or directory

Currently, most of the packages that are available in Parabola are not built by Parabola but come instead from a different Arch Linux distribution, depending on the architecture:

Architecture Upstream
armv7h Arch Linux ARM
i686 Arch Linux 32
x86_64 Arch Linux

Parabola developers then have a blacklist package that blacklists all the known Arch Linux nonfree packages, and also has its own packages in abslibre.

When the various Arch Linux GNU/Linux distributions update some libraries, we often don't manage to recompile the packages that depend on them in time, for various reasons (packages not building, not enough time, not aware of it, etc). That create errors like the mplayer error mentioned above.

2 Approaches to solve the issue

This article is a stub.
This typically means the article is a placeholder for more content to come. Knowledgeable users are encouraged to help expand the article.

This section will document the various methods that have been used over time, with their advantages and disadvantages.

3 Howto

3.1 Enable packages to use compatibility libraries

This will explain how to convert packages to be able to use compatibility libraries. For instance if you convert the 'openttd' package to use icu compatibility libraries.

Once you converted the 'openttd' package, let's suppose that at the time of building, The most recent icu version is 67 and that it is available in Parabola. When icu 68 comes in parabola, even if there is no icu 67 compatibility package, you will be able to make one afterward, without needing to recompile 'openttd'.

3.1.1 Libraries meeting the requirements

The following package and libraries meets the requirements:

  • icu

3.1.2 Making sure the library meets the requirements

First you need to make sure that the library you target (here icu) meets some requirements.

If it's listed in the previous section, you can skip this section.

If not you'll need to do some checks.

As the ICU package and library meets all the requirements to enable to use compatibility libraries, so we'll use it as an example.

First we need to check if the package provides some libraries:

$ pacman -Q -i icu
[...]
Provides        : libicudata.so=67-32  libicui18n.so=67-32  libicuio.so=67-32  libicutest.so=67-32   libicutu.so=67-32  libicuuc.so=67-32
[...]

Here it does, so we're good.

We need to do that check that for every architecture supported by Parabola as the upstream Arch Linux distributions are often not synchronized with each other: some are slower to adapt the last changes from Arch Linux. If we don't, then the architectures that don't have the 'Provides' will most likely not be able to build the packages you convert anymore.

Then we can see if the library is versioned:

$ nm -D --demangle --with-symbol-versions /usr/lib/libicui18n.so.65.1
[...]
001f3960 T icu_65::BucketList::~BucketList()@@Base
[...]
Here we can see that some symbols have icu_65 in front, so we're good.

If the library is not versioned, it could lead to strange bugs or crashes if some symbols changed in some slight incompatible way.

3.1.3 Converting packages (long version)

If you don't care about understanding how it works, you can skip to the next section which sumarize it a lot.

In this section we will go into the conversion process in more details to understand what we are doing.

We will take the example of openttd to do the conversion.

First look at the dependencies, here before the conversion we had:

depends=('libpng' 'sdl' 'icu' 'fontconfig' 'lzo' 'hicolor-icon-theme' 'desktop-file-utils' 'xz' 'fluidsynth')

And now we have:

depends=('libpng' 'sdl' 'fontconfig' 'lzo' 'hicolor-icon-theme' 'desktop-file-utils' 'xz' 'fluidsynth')
depends+=('libicui18n.so' 'libicuuc.so')

So let's look at it in more details. First 'libicui18n.so' and 'libicuuc.so' were added in the dependencies. When building the package: when we look at the resulting package, we 'libicui18n.so' became 'libicui18n.so=67-32' and 'libicuuc.so' became 'libicuuc.so=67-32'. So during the build, the exact versions were filled automatically:

$ pacman -Q -i openttd
Depends On      : libpng  sdl  fontconfig  lzo  hicolor-icon-theme  desktop-file-utils  xz  fluidsynth   libicui18n.so=67-32  libicuuc.so=67-32
Optional Deps   : openttd-opengfx: free graphics

Here we can also see that 'icu' is now missing from the dependencies: if openttd depended on 'icu', another package (like icu-67-compat, icu-parabola, etc) could not be used instead of 'icu'. Here any package that provides 'libicui18n.so=67-32' and 'libicuuc.so=67-32' can be used'

And currently 'icu' provides that:

$ pacman -Q -i icu
[...]
Provides        : libicudata.so=67-32  libicui18n.so=67-32  libicuio.so=67-32  libicutest.so=67-32  libicutu.so=67-32  libicuuc.so=67-32

But when Arch Linux will have icu-68, 'libicui18n.so=67-32' and 'libicuuc.so=67-32' could then be satisfied by another package, like icu-67-compat.

Then you might wonder how to find which libraries you need to add. If you're unsure you can add them all:

depends+=('libicudata.so' 'libicui18n.so' 'libicuio.so' 'libicutest.so' 'libicutu.so' 'libicuuc.so')

Near the end of the build, in the packaging, makepkg should output some warning(s) that tells you which ones weren't necessary. So if you're in a hurry you can add them all, and if not you can just build once and remove the unnecessary libraries and build a second time.

Another way that is more complicated but doesn't require building twice is to look at the binaries of the package (here openttd) to see which library they use.

So if we look at openttd, there is a single binary. Usually packages have several binaries and libraries in various places.

If we use ldd, it will give us 3 libraries:

$ ldd /usr/bin/openttd | grep icu
  libicui18n.so.67 => /usr/lib/libicui18n.so.67 (0xf661b000)
  libicuuc.so.67 => /usr/lib/libicuuc.so.67 (0xf6435000)
  libicudata.so.67 => /usr/lib/libicudata.so.67 (0xf423e000)

That looks strange as we only added two before. In reality, ldd not only gives the direct dependencies of /usr/bin/openttd, but it also gives all the indirect dependencies. Here /usr/bin/openttd depends on 'libicuuc.so.67' which in turn depends on 'libicudata.so.67', so this is why we see 3 libraries here.

If we inspect /usr/bin/openttd with readelf instead, we can see the direct dependencies:

$ readelf -d /usr/bin/openttd 

Dynamic section at offset 0x6257b8 contains 41 entries:

 Tag        Type                         Name/Value
0x00000001 (NEEDED)                     Shared library: [libpthread.so.0]
0x00000001 (NEEDED)                     Shared library: [libSDL-1.2.so.0]
0x00000001 (NEEDED)                     Shared library: [libz.so.1]
0x00000001 (NEEDED)                     Shared library: [liblzma.so.5]
0x00000001 (NEEDED)                     Shared library: [liblzo2.so.2]
0x00000001 (NEEDED)                     Shared library: [libpng16.so.16]
0x00000001 (NEEDED)                     Shared library: [libfontconfig.so.1]
0x00000001 (NEEDED)                     Shared library: [libfreetype.so.6]
0x00000001 (NEEDED)                     Shared library: [libicui18n.so.67]
0x00000001 (NEEDED)                     Shared library: [libicuuc.so.67]
0x00000001 (NEEDED)                     Shared library: [libfluidsynth.so.2]
0x00000001 (NEEDED)                     Shared library: [libstdc++.so.6]
0x00000001 (NEEDED)                     Shared library: [libm.so.6]
0x00000001 (NEEDED)                     Shared library: [libc.so.6]
0x00000001 (NEEDED)                     Shared library: [libgcc_s.so.1]
[...]

We can also verify that 'libicuuc.so.67' depends on 'libicudata.so.67' with the same command:

$ readelf -d /usr/lib/libicuuc.so.67

Dynamic section at offset 0x1e2cbc contains 35 entries:
  Tag        Type                         Name/Value
 0x00000001 (NEEDED)                     Shared library: [libicudata.so.67]
 0x00000001 (NEEDED)                     Shared library: [libpthread.so.0]
 0x00000001 (NEEDED)                     Shared library: [libdl.so.2]
 0x00000001 (NEEDED)                     Shared library: [libstdc++.so.6]
 0x00000001 (NEEDED)                     Shared library: [libm.so.6]
 0x00000001 (NEEDED)                     Shared library: [libgcc_s.so.1]
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]
 0x00000001 (NEEDED)                     Shared library: [ld-linux.so.2]
 0x0000000e (SONAME)                     Library soname: [libicuuc.so.67]

After having taken care of the dependencies, there is one one more things to take care of: Many PKGBUILDS in abslibre have a package() function that looks like that:

package() {
  local _icu_ver
  _icu_ver=$(pacman -S --print-format='%v' icu)
  depends+=("icu>=${_icu_ver}" "icu<$((${_icu_ver%%.*} + 1))")

  cd ${pkgname}-${pkgver} 
  [...]
  make install
}

That was introduced previously when 'icu' didn't have anything in 'Provides'. It made it impossible to upgrade a given package (like openttd) if Arch Linux upgraded the 'icu' package to a newer incompatible version. This way you could not upgrade, but at least openttd wasn't broken and did continue to work.

With this new system we can upgrade to newer ICU, while keeping packages like openttd working, so we don't need it anymore.

So we just have to remove the following 3 lines:

local _icu_ver
_icu_ver=$(pacman -S --print-format='%v' icu)
depends+=("icu>=${_icu_ver}" "icu<$((${_icu_ver%%.*} + 1))")

See the next section for a summary

3.1.4 Converting packages (summary)

In a nutshell, for packages that depends on 'icu':

First remove 'icu' from the list of dependencies.

Then a line with the libraries that the package depends on, for instance:

depends+=('libicui18n.so' 'libicuuc.so')

If you have more libraries than necessary, for instance:

depends+=('libicudata.so' 'libicui18n.so' 'libicuio.so' 'libicutest.so' 'libicutu.so' 'libicuuc.so')

it's not a big issue. It will work and makepkg will just output a warning telling you which library to remove from the list.

Then in the package() function(s), remove the following lines:

local _icu_ver
_icu_ver=$(pacman -S --print-format='%v' icu)
depends+=("icu>=${_icu_ver}" "icu<$((${_icu_ver%%.*} + 1))")

3.2 Create a compatibility package for a given library

This article is a stub.
This typically means the article is a placeholder for more content to come. Knowledgeable users are encouraged to help expand the article.

3.2.1 Create a compatibility package (summary)

This article is a stub.
This typically means the article is a placeholder for more content to come. Knowledgeable users are encouraged to help expand the article.

First Copy the package from Arch Linux git repository.

Then Change the package name to something like 'icu-67-compat'.

Add new architectures if necessary.

Then, add the following lines:

# require the main ICU version always be > this one
depends+=("icu>$pkgver")
conflicts=("icu<=$pkgver")

In package(), after the usual commands to install the library that typically looks like that:

package() {
  cd icu/source
  make -j1 DESTDIR="${pkgdir}" install

make sure that the license is installed:

 # Install license
 install -Dm644 "${srcdir}"/icu/LICENSE "${pkgdir}/usr/share/licenses/${pkgname}/LICENSE"

And remove all the files that could conflict with the Arch Linux package for that Library:

 # to avoid conflicts against the preferred 'icu' package, we remove all files
 # except the actual shared libraries
 rm -r "${pkgdir}"/usr/{bin,include,lib/{icu,pkgconfig},share}
 rm "${pkgdir}"/usr/lib/*.so

3.3 Depend on a specific library version

This article is a stub.
This typically means the article is a placeholder for more content to come. Knowledgeable users are encouraged to help expand the article.

When it's not possible to use compatibility versions, for instance if the library package has no 'Provides', we could resort to depending on an exact version of the package of a library.

3.3.1 Depend on a specific library version (summary)

This article is a stub.
This typically means the article is a placeholder for more content to come. Knowledgeable users are encouraged to help expand the article.

Add the package of the library in the dependencies if it's not already there:

depends+=('icu')

Then in the package() function(s), add the following lines:

local _icu_ver
_icu_ver=$(pacman -S --print-format='%v' icu)
depends+=("icu>=${_icu_ver}" "icu<$((${_icu_ver%%.*} + 1))")