Hacking:mksource()

From ParabolaWiki
Jump to: navigation, search

mksource() is a non-standard Parabola extension to the (libre-)makepkg program. It is a mechanism to create pre-cleaned source-balls, for distribution in the Parabola repos, in accordance with the GNU Free System Distribution Guidelines (FSDG), from upstream sources which require some FSDG-fitness treatment. Otherwise, Parabola could not publish all source packages.

This is accomplished, by moving all patching out of the configure() function, and into a mksource() function instead, and moving the URLs of upstream sources which require FSDG-fitness patching, into a mksource() array, replacing them in the standard source=() array, with a URL pointing to the Parabola repos, where the pre-cleaned source-balls are (or will be) accessible. In the chicken-and-egg case where the libre sources do not yet exist, the mksource() mechanism is implicitly activated to create them, before the standard (libre-)makepkg program resumes normally.

The reason why there must be some special handling to accomplish this, is because the default behavior, is such that the source-ball gets rolled before prepare() runs. If that happened after prepare() runs, the mksource() extension would not be needed. Unfortunately, although it was introduced several years ago, it's application still is far from comprehensive. Very few Parabola PKGBUILDs actually use it; although almost every PKGBUILD in [libre] probably should, if it has any value at all.

There is an on-going discussion about mksource(); to converge on a more elegant, less intrusive way to produce pre-cleaned source-balls. The current favored proposal, is to remove the mksource() mechanism entirely, and modify (libre-)makepkg to roll the source-ball after prepare(), rather than before. Regardless, mksource() works well; so there is no imperative to re-implement. It is relatively easy to create a mksource PKGBUILD.


1 Creating a mksource PKGBUILD

  1. add a mksource() function to the PKGBUILD, and populate it with all liberation steps which otherwise would be in prepare()
  2. if the above step results in an empty prepare(), delete prepare()
  3. prefix the 'source', checksums, and 'validpgpkeys' array(s) with 'mk' (eg: 'mksha256sums')
  4. restore those standard arrays above with data corresponding to a libre source-ball and it's signature, with URLs of the form: repo.parabola.nu/other/<PKGNAME>-libre/<PKGNAME>-<PKGVER>-parabola.tar.gz
  5. if the libre source-ball is not found on the parabola server, it is generated automatically per the mksource mechanism, before starting the normal build()
  6. otherwise, the build continues normally (bypassing the mksource() function)
  7. in either case, `librerelease` will ensure that the source-ball(s) are published

If the PKGBUILD is a libre replacement for a particularly gnarly PKGBUILD, in order to minimize the diff against the upstream(s), it is desirable to retain as many of the upstream LOCs as possible in tact, and over-ride them in a distinguished section farther down. The conventional over-ride clause below works in the simplest cases:

Given an upstream PKGBUILD:

 pkgname=foo
 pkgver=1.2.3
 source=(https://bithub.org/${pkgname}-${pkgver}.tar.gz{,.sig})
 sha256sums=(0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF SKIP)
 validpgpkeys=(0123456789ABCDEF0123456789ABCDEF01234567)

add after that:

 # parabola mksource over-rides
 mksource=(       ${source[*]}       )
 mksha256sums=(   ${sha256sums[*]}   )
 mkvalidpgpkeys=( ${validpgpkeys[*]} )
 source=(https://repo.parabola.nu/other/${pkgname}-libre/${pkgname}-${pkgver}-parabola.tar.gz{,.sig}
 sha256sums=( <LIBRE_SOURCE-BALL_SUM> SKIP )
 validpgpkeys=( <YOUR_GPG_KEY> )

<LIBRE_SOURCE-BALL_SUM> will be added later, after the libre source-ball is generated.


2 Program Flow

mksource() is a fairly complex and confusing mechanism. However, it is benign and uninteresting, in normal cases where the PKGBUILD has no mksource() function, or even in those special cases, where the libre source(s) already exist (either in the makepkg.conf::$SRCDEST directory, or on the Parabola repo server). This section is mainly to assist with debugging and future development. Packagers can skip reading this section.

The overview of program flow (very shallow and possibly imprecise), for the case where some libre source-ball does not already exist, and must be created, is as follows:

Given:

  • a PKGBUILD with a mksource() function, a mksource() array, and a 'repo.parabola.nu' URL in the source() array
  • the mksource() array contains URLS for all upstream sources which require FSDG treatment
  • the source() array has at least two entries: the source-ball, and it's signature; and both are URLs to 'repo.parabola.nu'
  • the libre source-ball for this package version, is missing
  • mk*sums() is populated with a correct checksum for each upstream source in the mksource() array (use: `librefetch -g` to generate them)
  • 'libretools' is installed on the host system
  • /etc/makepkg.d/librefetch.conf is sourced by makepkg.conf - (mentioned for completeness - the 'libretools' install hook configures this)

When:

  • $ sudo libremakepkg

2.1 Begin Standard Operation

/usr/bin/libremakepkg:

 -> LOG: "Initializing the chroot..."
 -> LOG: "Starting pre-build activities..."
 -> LOG: "Downloading sources..."
 -> CALL: /usr/lib/libretools/chroot/makechrootpkg.sh::download_sources()

/usr/lib/libretools/chroot/makechrootpkg.sh::download_sources():

 -> CALL: VerifyLibreSources (this logs the presence of any existing libre sources)
 -> CALL: /usr/bin/makepkg --allsource (w/ special ENV and in-chroot makepkg.conf)

/usr/bin/makepkg --allsource (where: $SOURCEONLY == 2):

 -> LOG: "Making package:"
 -> CALL: /usr/share/makepkg/source.sh::download_sources('allarch')

/usr/share/makepkg/source.sh::download_sources():

 NOTE: in this example, PKGBUILD::source[0] is the non-existing libre source-ball
 -> LOG: "Retrieving sources..."
 -> LOG: "Downloading PKGBUILD::source[0]"
 -> (/usr/bin/librefetch is launched indirectly as pacman's download agent)

/usr/bin/librefetch::doit():

 NOTE: $inmirror signifies a parabola URL, not the existence of the file
 NOTE: <SPECIAL> scripts are modified in-memory, to perform the mksource magic
 -> 'download' mode ($src == PKGBUILD::source[0] , $inmirror == true)
   -> if found:
     NOTE: normal non-mksource cases, or libre source-ball already present
     -> return to /usr/bin/makepkg
   -> if not found:
     -> LOG: "curl: (22) The requested URL returned error: 404"
     -> 'create' mode
     -> CALL: /tmp/<SPECIAL>/makepkg -p /tmp/<SPECIAL>/PKGBUILD

2.2 Create the Libre Source-ball

/tmp/<SPECIAL>/makepkg (where: SOURCEONLY == 0):

 -> LOG: "Making source:"
 -> LOG: "Checking source dependencies..."
 -> CALL: /tmp/<SPECIAL>/source.sh::download_sources('allarch')

/tmp/<SPECIAL>/source.sh::download_sources():

 NOTE: in this example, PKGBUILD::mksource[0] is the upstream source-ball
       in practice, this is iterating over all mksource() entries
 -> LOG: "Retrieving sources..."
 -> LOG: "Downloading PKGBUILD::mksource[0]"
 -> (again, /usr/bin/librefetch is launched as download agent)

/usr/bin/librefetch::doit():

 -> 'download' mode ($src == PKGBUILD::mksource[0] , $inmirror == false)
 -> return to /tmp/<SPECIAL>/makepkg

/tmp/<SPECIAL>/makepkg (post-return from /usr/bin/librefetch):

 NOTE: all standard PKGBUILD functions are stubbed out of this special PKGBUILD
       it's single purpose is to run the real PKGBUILD::mksource() function
       the standard functions in the real PKGBUILD will run later in the chroot
 -> LOG: "Validating source files"
 -> LOG: "Extracting sources..."
 -> LOG: "Starting build()"
   -> LOG: "Starting mksource()"
   -> CALL: PKGBUILD::mksource()
 -> LOG: "Creating package"
 -> LOG: "Finished making:"
Note: Here we rewind back to /usr/share/makepkg/source.sh::download_sources() to get/create source[1] (the source-ball signature)

2.3 Create the Libre Source-ball Signature

/usr/share/makepkg/source.sh::download_sources():

 NOTE: in this example, PKGBUILD::source[1] is the non-existing libre source-ball signature
 -> LOG: "Retrieving sources..."
 -> LOG: "Downloading PKGBUILD::mksource[1]"
 -> (/usr/bin/librefetch is launched indirectly as pacman's download agent)

/usr/bin/librefetch::doit():

 -> 'download' mode ($src == PKGBUILD::source[1] , $inmirror == true)
   -> if found:
     NOTE: normal non-mksource cases
     -> rewind to /usr/bin/makepkg --allsource, as the note below
   -> if not found:
     NOTE: special mksource cases
     -> LOG: "curl: (22) The requested URL returned error: 404"
     -> 'create' mode ($src == PKGBUILD::source[1] , $inmirror == true)
       -> CALL: /usr/bin/librefetch::create_signature()

/usr/bin/librefetch::create_signature():

 -> LOG: "Signing package..."
 -> (prompts local user for GPG signature)
 -> LOG: "Created signature file"
Note: Here we rewind back to /usr/bin/makepkg --allsource

/usr/bin/makepkg:

 -> CALL: /usr/share/makepkg/integrity.sh::check_source_integrity()
   -> CALL: /usr/share/makepkg/integrity/verify_signature.sh
     -> LOG: "Verifying source file signatures with gpg"
   -> CALL: /usr/share/makepkg/integrity/verify_checksum.sh
     -> LOG: "Validating source files"
     -> if the standard *sums() checksums do not all match:
       -> LOG: "ERROR: One or more files did not pass the validity check!"
Note: Finally, we return to /usr/lib/libretools/chroot/makechrootpkg.sh::download_sources(), which can detect the validation failure above (which is expected in the special case as this example, in which a libre source-ball was just created), and attempt to remedy the situation gracefully. In such cases, the correct checksums will be injected into the abslibre PKGBUILD, the process will fork a new instance of the original calling process (usually libremakepkg) to handle the modified abslibre PKGBUILD, and the current process will terminate immediately. Otherwise in the normal cases, in which all checksums match, makechrootpkg.sh simply returns to the original /usr/bin/libremakepkg process, which commences to building the package in the normal way

/usr/lib/libretools/chroot/makechrootpkg.sh::download_sources():

 NOTE: the program will terminate here, if there is any problem with the libre sources,
         for example, if validation failed in the previous call to verify_checksum.sh(),
         which is to be expected, if the libre source-ball was created only just now
       this second call to VerifyLibreSources will handle the special situation gracefully,
         provided that all libre sources are in tact
       if VerifyLibreSources detects some fatal problem with the libre sources,
         the entire program will terminate with: "Could not download sources",
         otherwise, it will fork first, as described above
 -> CALL: VerifyLibreSources (compare libre sources with the initial call)
   -> if pass:
     NOTE: normal non-mksource cases
     -> return to /usr/bin/libremakepkg
   -> if fail:
     -> if the libre sources are in tact:
     NOTE: again, this is expected, in special mksource cases, such as this example
     -> LOG: "Running `updpkgsums` to account for the newly created libre source-ball(s)"
     -> LOG: "Generating checksums for source files..."
     -> LOG: =================================================
     -> LOG: | Restarting libremakepkg to complete the build |
     -> LOG: =================================================
     -> FORK: /usr/bin/libremakepkg <ORIGINAL_PARAMS>
     -> DIE
   -> else:
     -> DIE "Could not download sources"

2.4 Resume Standard Operation

/usr/bin/libremakepkg:

 -> if this is a fresh process, per VerifyLibreSources above:
   -> LOG: "Initializing the chroot..."
   -> LOG: "Starting pre-build activities..."
   -> LOG: "Downloading sources..."
 -> LOG: "Starting to build the package"
Note: From here on, the build/packaging process procedes as normal