Bootstrappable Debian

Wookey

Debconf 11, July 2011

What’s this about

New ports - very hard to bootstrap
Debian can’t build itself
Cyclic dependencies
Cross-building

Nomenclature

Build and Host
Bootstrap/staged
Install-deps/build-deps

Basics

Not cross-specific (in theory)
Very cross-related (in practice)
Always some cross, then some native.
- Toolchain
- Base system
- All canonical-node packages

Why should we care?

New ports
- 17 times so far
Optimised builds
Updating a lagged arch (m68k)
Non self-hosted arches (e.g avr32)
Debian ought not to need gentoo, OE

What currently happens

Start with OE build or gentoo base image
Hack, build, hack, build, bodge, build
Re-root, hack a bit less, build, more hacking and building
Manual and labour-intensive (xdeb helps somewhat)

Why crossing helps

Removes many doc- and tool- related cycles

Cycle analysis

Loops in Squeeze, Wheezy, Maverick, Natty
Install-deps matter too

How that was done

xdeb + graphviz
Scripts are in xdeb as ‘findcycles’

xdeb —generate-graph —apt-source | sccmap | gvpr -f make_strict.g | gvpr -f colour_nodes.g

Examples

qt -> poppler -> cups -> qt4-x11
krb5 <-> openldap
lapack <-> atlas
gettext -> libcroco -> glib2
webkit -> libsoup -> libproxy
udev -> gobject-introspection -> cairo -> mesa -> udev

Statistics - Squeeze

Analysed 9274 packages (2vcard to multitail)
2231 (24%) have build-dep loops
2186 - glib2.0
2000 - gobject-introspection
1866 - poppler/krb5
192 - webkit
67 - lapack
16 - lvm

Statistics - Ubuntu (maverick/natty)

Analysed 2240 packages (main)
706 (31%) have loops
610 - gobject-introspection (natty only)
495 - webkit/poppler/krb5
27 - network-manager
17 - lvm

Mechanism

Altered build-deps
Build-Depends-stage1, Build-depends-stage2
dpkg-dev fixed to recognise DEB_BUILD_OPTIONS=stage1
(dpkg-buildpackage, dpkg-checkbuilddeps, dpkg-source)
Package build modifed
Staged packages are marked with Buildstage: stage1

How intrusive is this?

Probably about 40 (?) packages affected
Patches generally not intrusive
Doing DEB_BUILD_OPTS=nodocs will add more

Typical patch (debhelper) (krb5)

+ifneq (,$(filter stage1,$(DEB_BUILD_OPTIONS)))
+   LDAP_DEP=
+   export DH_OPTIONS=-Nkrb5-kdc-ldap
+else
+   LDAP_DEP=--with-ldap
+endif

-    --enable-shared --with-ldap --without-tcl \
+    --enable-shared  --without-tcl $(LDAP_DEP)\

+ifneq (,$(filter stage1,$(DEB_BUILD_OPTIONS)))
    mv $(CURDIR)/debian/tmp/usr/lib/libkdb_ldap* \
            $(CURDIR)/debian/tmp/usr/lib/krb5/
    rm -f $(CURDIR)/debian/tmp/usr/lib/krb5/libkdb_ldap*.so
+endif

Typical patch (CDBS) (poppler)

+export DH_OPTIONS = -Nlibpoppler-qt2 -Nlibpoppler-qt-dev
-Nlibpoppler-qt4-3 -Nlibpoppler-qt4-dev
+
+DEB_ARCH_PACKAGES := libpoppler5 libpoppler-dev libpoppler-glib4 \
+ libpoppler-glib-dev poppler-utils poppler-dbg
+
+ifneq (,$(filter stage1,$(DEB_BUILD_OPTIONS)))
+# disable gtk stuff to minimize Build-Depends
+DEB_CONFIGURE_EXTRA_FLAGS +=                   \
+       --enable-libjpeg                        \
+       --disable-openjpeg                      \
+       --enable-poppler-glib                   \
+       --disable-poppler-qt                    \
+       --disable-poppler-qt4                   \
+       --enable-a4-paper                       \
+       --disable-gtk-doc                       \
+       --disable-gtk-test

Automated bootstrap

Analyse dependency tree
Break loop - do stage1 build
Build depending packages
Do proper build
Gustavo Alkmim working on this

Continuous integration

Stages will tend to rot
Autobuilder giving FTSBFS

Bootstrapping spec

http://wiki.debian.org/DebianBootstrap (like steve said)
http://wiki.debian.org/CircularBuildDependencies

cross debug tag

cross, crossbuild@debian.org
http://bugs.debian.org/cgi-bin/pkgreport.cgi?users=crossbuild@debian.org&tag=cross

New docs appearing

https://wiki.linaro.org/Platform/DevPlatform/CrossCompile/CrossBuilding
https://wiki.linaro.org/Platform/DevPlatform/CrossCompile/CrossbuildingQuickStart

Auto crossbuilder

For cross-FTBFS status

What do I need from you

Agreement this is worth doing
Should we make it policy?
Feedback on mechanism details
Package-maintainer help for loop-breaking
Help with autobuilder setup