Poudriere: 12-stable, running inside jail

Created on 10 Dec 2018  ·  17Comments  ·  Source: freebsd/poudriere

I'm having trouble running poudriere inside a jail since upgrading to 12-stable a couple of days ago. Without any changes to my config, I now get jail: IPv4 addresses clashwhile the poudriere builder jail is started during a build:

# poudriere bulk -j 12amd64 -p local -z zs64 -f /root/11amd64-local-zs64-pkglist
[00:00:00] Creating the reference jail... done
[00:00:01] Mounting system devices for 12amd64-local-zs64
[00:00:01] Mounting ports/packages/distfiles
[00:00:01] Using packages from previously failed build
[00:00:01] Mounting ccache from: /var/cache/ccache
[00:00:01] Mounting packages from: /p/data/packages/12amd64-local-zs64
[00:00:02] Appending to make.conf: /usr/local/etc/poudriere.d/make.conf
[00:00:02] Appending to make.conf: /usr/local/etc/poudriere.d/zs64-make.conf
/etc/resolv.conf -> /p/data/.m/12amd64-local-zs64/ref/etc/resolv.conf
[00:00:02] Starting jail 12amd64-local-zs64
jail: IPv4 addresses clash
[00:00:02] Cleaning up
[00:00:02] Unmounting file systems

I've fiddled with LOIP4 and LOIP6 and the outside jail IP configuration to no avail.

Inspecting the code in common.sh, it appears that poudriere tries to determine which IPs to pass to the jail start invocation, namely, ip4.addr and ip6.addr. I've hard-reset localipargs to the empty string (just after the case $IPS block at around line 7734), which makes my builds work correctly again.

Is this a potential incompatibility with 12-stable?

Base Issue

Most helpful comment

I've managed to get a working setup again, by using VIMAGE and a vnet jail. If the "classic" shared IP setup can be made to work, I would like to hear how.

All 17 comments

Dang, even with that patch, Python and Ruby fail to build.

Python:

checking getaddrinfo bug... yes
Fatal: You must get working getaddrinfo() function.
       or you can specify "--disable-ipv6".
===>  Script "configure" failed unexpectedly.

Ruby:

compiling raddrinfo.c
raddrinfo.c:214:17: warning: implicit declaration of function 'parse_numeric_port' is invalid in C99 [-Wimplicit-function-declaration]
    if (node && parse_numeric_port(service, &port)) {
                ^
1 warning generated.
compiling ifaddr.c
compiling getaddrinfo.c
In file included from getaddrinfo.c:86:
./addrinfo.h:165:12: error: conflicting types for 'getnameinfo'
extern int getnameinfo __P((
           ^
/usr/include/netdb.h:251:6: note: previous declaration is here
int             getnameinfo(const struct sockaddr *, socklen_t, char *,
                ^
getaddrinfo.c:408:7: warning: add explicit braces to avoid dangling else [-Wdangling-else]
                                } else if (strcmp(sp->s_proto, "tcp") == 0) {
                                  ^
1 warning and 1 error generated.
*** Error code 1

I've managed to get a working setup again, by using VIMAGE and a vnet jail. If the "classic" shared IP setup can be made to work, I would like to hear how.

Building ruby inside a 12.0-RELEASE-p2 jail still failing.
Building ruby inside an 11.2-RELEASE-p8 jail works.
Any insight?

Attached is the build log:
ruby-2.5.3_1,1.log

@stblassitude What did you have to modify to make it work? Specifically where is the vnet option needed? In the jail Poudriere is in or in Poudriere-created jails?

I think a workaround could be to set a unique loopback address:

LOIP4=127.0.0.2
LOIP6=::2

@bdrewery , using the settings you suggested generates the same warnings and errors:
(also tried LOIP4="127.0.0.2/32"). Seems like LOIP4 is being ignored. Should it be set elsewhere?
Setting the loopback of the poudriere jail externally also fails.

grep '^[A-Z]' /usr/local/etc/poudriere.conf

LOIP4=127.0.0.2
LOIP6=::2
NO_ZFS=yes
FREEBSD_HOST=ftp://ftp.freebsd.org
RESOLV_CONF=/etc/resolv.conf
BASEFS=/p
USE_PORTLINT=no
USE_TMPFS=no
DISTFILES_CACHE=/usr/ports/distfiles
NOLINUX=yes
ALLOW_MAKE_JOBS=yes

[00:00:00] Warning: No loopback address defined, consider setting LOIP6/LOIP4 or assigning a loopback address to the jail.
[00:00:00] Updating portstree "default" with portsnap...Looking up portsnap.FreeBSD.org mirrors... 6 mirrors found.
In file included from getaddrinfo.c:86:
./addrinfo.h:165:12: error: conflicting types for 'getnameinfo'
extern int getnameinfo __P((
           ^
/usr/include/netdb.h:251:6: note: previous declaration is here
int             getnameinfo(const struct sockaddr *, socklen_t, char *,
                ^
--- ext/cgi/escape/all ---
--- escape.o ---
compiling escape.c
--- ext/ripper/all ---
--- pre-install-rb-default ---
installing default ripper libraries
--- ../../.ext/common/ripper.rb ---
--- ext/json/all ---
--- ../../.ext/common/json/add/rational.rb ---
--- ext/fiddle/all ---
--- fiddle.o ---
--- ext/openssl/all ---
--- ossl.o ---
--- ext/fiddle/all ---
compiling fiddle.c
--- ext/openssl/all ---
compiling ossl.c
--- ext/rbconfig/sizeof/all ---
--- limits.o ---
compiling limits.c
--- ext/socket/all ---
getaddrinfo.c:408:7: warning: add explicit braces to avoid dangling else [-Wdangling-else]
                                } else if (strcmp(sp->s_proto, "tcp") == 0) {
                                  ^
--- ext/bigdecimal/all ---
--- ../../.ext/common/bigdecimal/math.rb ---
--- ext/socket/all ---
1 warning and 1 error generated.
*** [getaddrinfo.o] Error code 1

@bdrewery , using the settings you suggested generates the same warnings and errors:
(also tried LOIP4="127.0.0.2/32"). Seems like LOIP4 is being ignored. Should it be set elsewhere?

It's because of this code in common.sh:

# If in a nested jail we may not even have a loopback to use.            
if [ ${JAILED} -eq 1 ]; then                                             
        # !! Note these exit statuses are inverted                       
        ifconfig | \                                                     
            awk -vip="${LOIP6}" '$1 == "inet6" && $2 == ip {exit 1}' && \
            LOIP6=                                                       
        ifconfig | \                                                     
            awk -vip="${LOIP4}" '$1 == "inet" && $2 == ip {exit 1}' && \ 
            LOIP4=                                                       
fi                                                                       

Try removing that for now. It's trying to ensure the current jail has the IP you're trying to assign to the nested jail. (The regression in question.)

When commenting that out, the following is displayed and no building proceeds:

jail: jail_set: Operation not permitted

Just to confirm, I'm running Poudriere inside an iocage jail on a host running FreeBSD 12.0-RELEASE-p3 GENERIC

# iocage list -l
+------+------------+------+-------+------+-----------------+--------------------+-----+----------+----------+
| JID  |    NAME    | BOOT | STATE | TYPE |     RELEASE     |        IP4         | IP6 | TEMPLATE | BASEJAIL |
+======+============+======+=======+======+=================+====================+=====+==========+==========+
| 2963 | poudriere  | on   | up    | jail | 12.0-RELEASE-p3 | lo1|192.168.250.10 | -   | -        | no       |

Compilation of ruby fails for the jails listed below:

root@poudriere:~ # poudriere jail -l
[00:00:00] Warning: No loopback address defined, consider setting LOIP6/LOIP4 or assigning a loopback address to the jail.
JAILNAME    VERSION         ARCH  METHOD TIMESTAMP           PATH
11_2        11.2-RELEASE-p9 amd64 ftp    2019-03-05 13:49:56 /p/jails/11_2
12_0        12.0-RELEASE-p3 amd64 ftp    2019-03-05 13:52:05 /p/jails/12_0

@stblassitude What did you have to modify to make it work? Specifically where is the vnet option needed? In the jail Poudriere is in or in Poudriere-created jails?

Here's my jail.conf:

mount.devfs;
devfs_ruleset = 4;

exec.clean;
exec.jail_user = "root";
exec.system_user = "root";
exec.consolelog = "/var/log/jail_${name}.log";
exec.prestart = "/root/bin/jail-helper prestart ${name} ${host.hostname} ${path}";
exec.poststop = "/root/bin/jail-helper poststop ${name} ${host.hostname} ${path}";
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
interface = "bridge0";
allow.raw_sockets;
allow.sysvipc;
...
pkg {
        path = "/jail/pkg.zs64.net";
        vnet;
        vnet.interface = "ji${name}";
        host.hostname = "pkg.zs64.net";
        children.max = 40;
        exec.prestart += "/root/bin/jail-helper prevnet ${name} bridge1";
        exec.poststart += "zfs jail $name data/jail/${host.hostname}/poudriere";
        exec.poststart += "jexec $name zfs mount -a";
        #exec.prestop += "zfs unjail $name data/jail/${host.hostname}/poudriere";
        exec.poststop += "/root/bin/jail-helper postvnet ${name}";
        allow.mount;
        allow.mount.devfs;
        allow.mount.linprocfs;
        allow.mount.nullfs;
        allow.mount.procfs;
        allow.mount.tmpfs;
        allow.mount.zfs;
        allow.socket_af;
        allow.chflags;
        enforce_statfs=1;
}

The helper script does this:

#!/bin/sh

#
# Mount system directories via nullfs
#

cmd="$1"
name="$2"
host="$3"
path="$4"

rofs="/bin /lib /libexec /sbin /usr/bin /usr/include /usr/lib /usr/libdata /usr/libexec /usr/sbin /usr/share /usr/ports"

mountall() {
    mount -t devfs -o ruleset=10 devfs ${path}/dev
    ls ${path}/dev/null >/dev/null
    for i in $rofs; do
        mount -t nullfs -o ro "${i}" "${path}${i}"
    done
    mount -t nullfs -w /freebsd/distfiles ${path}/var/ports/distfiles
}

umountall() {
    mount | \
        sed -nEe 's#.* on ('"${path}"'/[^ ]*) \(.*#\1#p' | \
        sort -r | \
        xargs -n1 umount
}

vnet_create() {
    ifname="$(/sbin/ifconfig epair create)"
    ifbase="${ifname%%a}"
    /sbin/ifconfig "${ifbase}a" up name "jo${name}" >/dev/null
    /sbin/ifconfig "${ifbase}b" name "ji${name}" >/dev/null
    /sbin/ifconfig "${host}" addm "jo${name}"
}

vnet_destroy() {
    /sbin/ifconfig "jo${name}" destroy 2>/dev/null || true
}

case ${cmd} in
    prestart)
        umountall
        mountall
        ;;
    prevnet)
        vnet_destroy
        vnet_create
        ;;
    poststop)
        umountall
        ;;
    postvnet)
        vnet_destroy
        ;;
esac

Inside the pkg jail, I use this poudriere.conf:

BASEFS=/usr/local/poudriere
CCACHE_DIR=/var/cache/ccache
DISTFILES_CACHE=/var/ports/distfiles
FREEBSD_HOST=https://download.FreeBSD.org
PKG_REPO_SIGNING_KEY=/usr/local/etc/poudriere.key
TMPFS_LIMIT=6
URL_BASE=http://pkg.example.com
ZPOOL=data
ZROOTFS=/jail/pkg.example.com/poudriere

The ZFS datasets look like this inside the jail:

$ zfs list
NAME                                             USED  AVAIL  REFER  MOUNTPOINT
data                                            1.74T   858G    88K  /data
data/jail                                       1.33T   858G    96K  /jail
data/jail/pkg.example.com                          34.2G   858G  5.54G  /jail/pkg.example.com
data/jail/pkg.example.com/poudriere                24.3G   858G    88K  /p
data/jail/pkg.example.com/poudriere/data           19.1G   858G    96K  /p/data
data/jail/pkg.example.com/poudriere/data/.m        11.6G   858G   112K  /p/data/.m
data/jail/pkg.example.com/poudriere/data/cache      260M   858G  23.8M  /p/data/cache
data/jail/pkg.example.com/poudriere/data/logs      3.17G   858G  2.74G  /p/data/logs
data/jail/pkg.example.com/poudriere/data/packages  4.12G   858G  1.95G  /p/data/packages
data/jail/pkg.example.com/poudriere/data/wrkdirs     88K   858G    88K  /p/data/wrkdirs
data/jail/pkg.example.com/poudriere/jails          1.27G   858G    88K  /p/jails
data/jail/pkg.example.com/poudriere/jails/11amd64    88K   858G    88K  /usr/local/poudriere/jails/11amd64
data/jail/pkg.example.com/poudriere/jails/12amd64  1.27G   858G  1.24G  /usr/local/poudriere/jails/12amd64
data/jail/pkg.example.com/poudriere/ports          3.91G   858G    88K  /p/ports
data/jail/pkg.example.com/poudriere/ports/local    3.91G   858G   903M  /usr/local/poudriere/ports/local

I believe I had this issue as well, but since I'd tried it for the first time on 12, I didn't think it was related to a 12 upgrade. Just looking on to see what solution you all come up with here.

I'm found test case, build lang/python27 or lang/python3 with IPv6 option and dual stack jail.

If poudriere jail export jail.conf
ip4=inherit; ip6=inherit;
build sucess

if export ip4.addr and ip6.addr adresses build fail on configure
checking for getaddrinfo... yes checking getaddrinfo bug... yes Fatal: You must get working getaddrinfo() function. or you can specify "--disable-ipv6".
disable option IPv6 fix build.

Apparently an error somewhere in set lo for IPv6 this now (::1/128) not /8

Interestingly, the difference is not what an interactive worker is.
If use bulk -i auto build fail configure, hand build in this jail success
make -C /usr/ports/lang/python27/ .... checking for getaddrinfo... yes checking getaddrinfo bug... no checking for getnameinfo... (cached) yes .... creating Makefile ....

RESTRICT_NETWORKING=no fix this issue, maybe add NO_RESTRICT_NETWORKING_PACKAGES="python27 ..." similar ALLOW_NETWORKING_PACKAGES ?

I've managed to get a working setup again, by using VIMAGE and a vnet jail. If the "classic" shared IP setup can be made to work, I would like to hear how.

Based on the above comment, I also went with a vnet solution. Worked for me. Thank you.

@stblassitude What did you have to modify to make it work? Specifically where is the vnet option needed? In the jail Poudriere is in or in Poudriere-created jails?

It is in the configuration for the poudriere jail. I did this.

For /etc/rc.conf:

cloned_interfaces="bridge0"
ifconfig_bridge0="addm ix0 up"

Then for the jail configuration:

$ sudo iocage set vnet=on pkg01
$ sudo iocage set ip4_addr="vnet0|10.55.0.29/24" pkg01
$ sudo iocage set ip6_addr="vnet0|[redacted]:23/64" pkg01

I'll have a blog post in the next week or so.

Just wanted to follow up with an update I made while running 12.1-RELEASE-p7 after reading this:

"It also means that when a jailed daemon attaches to a port on the “loopback address,” it’s actually attaching to the jail’s external IP address."
Lucas, Michael W. FreeBSD Mastery: Jails (IT Mastery Book 15)

Setting LOIP4 in poudriere.conf to the jail's ip4_addr allowed ruby to compile.

In my case:
# iocage get ip4_addr poudriere
lo1|192.168.10.100

# grep LOIP4 /usr/local/etc/poudriere.conf
LOIP4=192.168.10.100

Was this page helpful?
0 / 5 - 0 ratings