I'm having trouble running poudriere inside a jail since upgrading to 12-stable a couple of days ago. Without any changes to my config, I now get jail: IPv4 addresses clash
while the poudriere builder jail is started during a build:
# poudriere bulk -j 12amd64 -p local -z zs64 -f /root/11amd64-local-zs64-pkglist
[00:00:00] Creating the reference jail... done
[00:00:01] Mounting system devices for 12amd64-local-zs64
[00:00:01] Mounting ports/packages/distfiles
[00:00:01] Using packages from previously failed build
[00:00:01] Mounting ccache from: /var/cache/ccache
[00:00:01] Mounting packages from: /p/data/packages/12amd64-local-zs64
[00:00:02] Appending to make.conf: /usr/local/etc/poudriere.d/make.conf
[00:00:02] Appending to make.conf: /usr/local/etc/poudriere.d/zs64-make.conf
/etc/resolv.conf -> /p/data/.m/12amd64-local-zs64/ref/etc/resolv.conf
[00:00:02] Starting jail 12amd64-local-zs64
jail: IPv4 addresses clash
[00:00:02] Cleaning up
[00:00:02] Unmounting file systems
I've fiddled with LOIP4 and LOIP6 and the outside jail IP configuration to no avail.
Inspecting the code in common.sh
, it appears that poudriere tries to determine which IPs to pass to the jail start invocation, namely, ip4.addr
and ip6.addr
. I've hard-reset localipargs
to the empty string (just after the case $IPS
block at around line 7734), which makes my builds work correctly again.
Is this a potential incompatibility with 12-stable?
Dang, even with that patch, Python and Ruby fail to build.
Python:
checking getaddrinfo bug... yes
Fatal: You must get working getaddrinfo() function.
or you can specify "--disable-ipv6".
===> Script "configure" failed unexpectedly.
Ruby:
compiling raddrinfo.c
raddrinfo.c:214:17: warning: implicit declaration of function 'parse_numeric_port' is invalid in C99 [-Wimplicit-function-declaration]
if (node && parse_numeric_port(service, &port)) {
^
1 warning generated.
compiling ifaddr.c
compiling getaddrinfo.c
In file included from getaddrinfo.c:86:
./addrinfo.h:165:12: error: conflicting types for 'getnameinfo'
extern int getnameinfo __P((
^
/usr/include/netdb.h:251:6: note: previous declaration is here
int getnameinfo(const struct sockaddr *, socklen_t, char *,
^
getaddrinfo.c:408:7: warning: add explicit braces to avoid dangling else [-Wdangling-else]
} else if (strcmp(sp->s_proto, "tcp") == 0) {
^
1 warning and 1 error generated.
*** Error code 1
I've managed to get a working setup again, by using VIMAGE and a vnet jail. If the "classic" shared IP setup can be made to work, I would like to hear how.
Building ruby inside a 12.0-RELEASE-p2 jail still failing.
Building ruby inside an 11.2-RELEASE-p8 jail works.
Any insight?
Attached is the build log:
ruby-2.5.3_1,1.log
@stblassitude What did you have to modify to make it work? Specifically where is the vnet
option needed? In the jail Poudriere is in or in Poudriere-created jails?
I think a workaround could be to set a unique loopback address:
LOIP4=127.0.0.2
LOIP6=::2
@bdrewery , using the settings you suggested generates the same warnings and errors:
(also tried LOIP4="127.0.0.2/32"). Seems like LOIP4 is being ignored. Should it be set elsewhere?
Setting the loopback of the poudriere jail externally also fails.
LOIP4=127.0.0.2
LOIP6=::2
NO_ZFS=yes
FREEBSD_HOST=ftp://ftp.freebsd.org
RESOLV_CONF=/etc/resolv.conf
BASEFS=/p
USE_PORTLINT=no
USE_TMPFS=no
DISTFILES_CACHE=/usr/ports/distfiles
NOLINUX=yes
ALLOW_MAKE_JOBS=yes
[00:00:00] Warning: No loopback address defined, consider setting LOIP6/LOIP4 or assigning a loopback address to the jail.
[00:00:00] Updating portstree "default" with portsnap...Looking up portsnap.FreeBSD.org mirrors... 6 mirrors found.
In file included from getaddrinfo.c:86:
./addrinfo.h:165:12: error: conflicting types for 'getnameinfo'
extern int getnameinfo __P((
^
/usr/include/netdb.h:251:6: note: previous declaration is here
int getnameinfo(const struct sockaddr *, socklen_t, char *,
^
--- ext/cgi/escape/all ---
--- escape.o ---
compiling escape.c
--- ext/ripper/all ---
--- pre-install-rb-default ---
installing default ripper libraries
--- ../../.ext/common/ripper.rb ---
--- ext/json/all ---
--- ../../.ext/common/json/add/rational.rb ---
--- ext/fiddle/all ---
--- fiddle.o ---
--- ext/openssl/all ---
--- ossl.o ---
--- ext/fiddle/all ---
compiling fiddle.c
--- ext/openssl/all ---
compiling ossl.c
--- ext/rbconfig/sizeof/all ---
--- limits.o ---
compiling limits.c
--- ext/socket/all ---
getaddrinfo.c:408:7: warning: add explicit braces to avoid dangling else [-Wdangling-else]
} else if (strcmp(sp->s_proto, "tcp") == 0) {
^
--- ext/bigdecimal/all ---
--- ../../.ext/common/bigdecimal/math.rb ---
--- ext/socket/all ---
1 warning and 1 error generated.
*** [getaddrinfo.o] Error code 1
@bdrewery , using the settings you suggested generates the same warnings and errors:
(also tried LOIP4="127.0.0.2/32"). Seems like LOIP4 is being ignored. Should it be set elsewhere?
It's because of this code in common.sh:
# If in a nested jail we may not even have a loopback to use.
if [ ${JAILED} -eq 1 ]; then
# !! Note these exit statuses are inverted
ifconfig | \
awk -vip="${LOIP6}" '$1 == "inet6" && $2 == ip {exit 1}' && \
LOIP6=
ifconfig | \
awk -vip="${LOIP4}" '$1 == "inet" && $2 == ip {exit 1}' && \
LOIP4=
fi
Try removing that for now. It's trying to ensure the current jail has the IP you're trying to assign to the nested jail. (The regression in question.)
When commenting that out, the following is displayed and no building proceeds:
jail: jail_set: Operation not permitted
Just to confirm, I'm running Poudriere inside an iocage jail on a host running FreeBSD 12.0-RELEASE-p3 GENERIC
# iocage list -l
+------+------------+------+-------+------+-----------------+--------------------+-----+----------+----------+
| JID | NAME | BOOT | STATE | TYPE | RELEASE | IP4 | IP6 | TEMPLATE | BASEJAIL |
+======+============+======+=======+======+=================+====================+=====+==========+==========+
| 2963 | poudriere | on | up | jail | 12.0-RELEASE-p3 | lo1|192.168.250.10 | - | - | no |
Compilation of ruby fails for the jails listed below:
root@poudriere:~ # poudriere jail -l
[00:00:00] Warning: No loopback address defined, consider setting LOIP6/LOIP4 or assigning a loopback address to the jail.
JAILNAME VERSION ARCH METHOD TIMESTAMP PATH
11_2 11.2-RELEASE-p9 amd64 ftp 2019-03-05 13:49:56 /p/jails/11_2
12_0 12.0-RELEASE-p3 amd64 ftp 2019-03-05 13:52:05 /p/jails/12_0
@stblassitude What did you have to modify to make it work? Specifically where is the
vnet
option needed? In the jail Poudriere is in or in Poudriere-created jails?
Here's my jail.conf
:
mount.devfs;
devfs_ruleset = 4;
exec.clean;
exec.jail_user = "root";
exec.system_user = "root";
exec.consolelog = "/var/log/jail_${name}.log";
exec.prestart = "/root/bin/jail-helper prestart ${name} ${host.hostname} ${path}";
exec.poststop = "/root/bin/jail-helper poststop ${name} ${host.hostname} ${path}";
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
interface = "bridge0";
allow.raw_sockets;
allow.sysvipc;
...
pkg {
path = "/jail/pkg.zs64.net";
vnet;
vnet.interface = "ji${name}";
host.hostname = "pkg.zs64.net";
children.max = 40;
exec.prestart += "/root/bin/jail-helper prevnet ${name} bridge1";
exec.poststart += "zfs jail $name data/jail/${host.hostname}/poudriere";
exec.poststart += "jexec $name zfs mount -a";
#exec.prestop += "zfs unjail $name data/jail/${host.hostname}/poudriere";
exec.poststop += "/root/bin/jail-helper postvnet ${name}";
allow.mount;
allow.mount.devfs;
allow.mount.linprocfs;
allow.mount.nullfs;
allow.mount.procfs;
allow.mount.tmpfs;
allow.mount.zfs;
allow.socket_af;
allow.chflags;
enforce_statfs=1;
}
The helper script does this:
#!/bin/sh
#
# Mount system directories via nullfs
#
cmd="$1"
name="$2"
host="$3"
path="$4"
rofs="/bin /lib /libexec /sbin /usr/bin /usr/include /usr/lib /usr/libdata /usr/libexec /usr/sbin /usr/share /usr/ports"
mountall() {
mount -t devfs -o ruleset=10 devfs ${path}/dev
ls ${path}/dev/null >/dev/null
for i in $rofs; do
mount -t nullfs -o ro "${i}" "${path}${i}"
done
mount -t nullfs -w /freebsd/distfiles ${path}/var/ports/distfiles
}
umountall() {
mount | \
sed -nEe 's#.* on ('"${path}"'/[^ ]*) \(.*#\1#p' | \
sort -r | \
xargs -n1 umount
}
vnet_create() {
ifname="$(/sbin/ifconfig epair create)"
ifbase="${ifname%%a}"
/sbin/ifconfig "${ifbase}a" up name "jo${name}" >/dev/null
/sbin/ifconfig "${ifbase}b" name "ji${name}" >/dev/null
/sbin/ifconfig "${host}" addm "jo${name}"
}
vnet_destroy() {
/sbin/ifconfig "jo${name}" destroy 2>/dev/null || true
}
case ${cmd} in
prestart)
umountall
mountall
;;
prevnet)
vnet_destroy
vnet_create
;;
poststop)
umountall
;;
postvnet)
vnet_destroy
;;
esac
Inside the pkg
jail, I use this poudriere.conf
:
BASEFS=/usr/local/poudriere
CCACHE_DIR=/var/cache/ccache
DISTFILES_CACHE=/var/ports/distfiles
FREEBSD_HOST=https://download.FreeBSD.org
PKG_REPO_SIGNING_KEY=/usr/local/etc/poudriere.key
TMPFS_LIMIT=6
URL_BASE=http://pkg.example.com
ZPOOL=data
ZROOTFS=/jail/pkg.example.com/poudriere
The ZFS datasets look like this inside the jail:
$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
data 1.74T 858G 88K /data
data/jail 1.33T 858G 96K /jail
data/jail/pkg.example.com 34.2G 858G 5.54G /jail/pkg.example.com
data/jail/pkg.example.com/poudriere 24.3G 858G 88K /p
data/jail/pkg.example.com/poudriere/data 19.1G 858G 96K /p/data
data/jail/pkg.example.com/poudriere/data/.m 11.6G 858G 112K /p/data/.m
data/jail/pkg.example.com/poudriere/data/cache 260M 858G 23.8M /p/data/cache
data/jail/pkg.example.com/poudriere/data/logs 3.17G 858G 2.74G /p/data/logs
data/jail/pkg.example.com/poudriere/data/packages 4.12G 858G 1.95G /p/data/packages
data/jail/pkg.example.com/poudriere/data/wrkdirs 88K 858G 88K /p/data/wrkdirs
data/jail/pkg.example.com/poudriere/jails 1.27G 858G 88K /p/jails
data/jail/pkg.example.com/poudriere/jails/11amd64 88K 858G 88K /usr/local/poudriere/jails/11amd64
data/jail/pkg.example.com/poudriere/jails/12amd64 1.27G 858G 1.24G /usr/local/poudriere/jails/12amd64
data/jail/pkg.example.com/poudriere/ports 3.91G 858G 88K /p/ports
data/jail/pkg.example.com/poudriere/ports/local 3.91G 858G 903M /usr/local/poudriere/ports/local
I believe I had this issue as well, but since I'd tried it for the first time on 12, I didn't think it was related to a 12 upgrade. Just looking on to see what solution you all come up with here.
I'm found test case, build lang/python27
or lang/python3
with IPv6 option and dual stack jail.
If poudriere jail export jail.conf
ip4=inherit;
ip6=inherit;
build sucess
if export ip4.addr
and ip6.addr
adresses build fail on configure
checking for getaddrinfo... yes
checking getaddrinfo bug... yes
Fatal: You must get working getaddrinfo() function.
or you can specify "--disable-ipv6".
disable option IPv6 fix build.
Apparently an error somewhere in set lo for IPv6 this now (::1/128) not /8
Interestingly, the difference is not what an interactive worker is.
If use bulk -i
auto build fail configure, hand build in this jail success
make -C /usr/ports/lang/python27/
....
checking for getaddrinfo... yes
checking getaddrinfo bug... no
checking for getnameinfo... (cached) yes
....
creating Makefile
....
RESTRICT_NETWORKING=no
fix this issue, maybe add NO_RESTRICT_NETWORKING_PACKAGES="python27 ..."
similar ALLOW_NETWORKING_PACKAGES
?
I've managed to get a working setup again, by using VIMAGE and a vnet jail. If the "classic" shared IP setup can be made to work, I would like to hear how.
Based on the above comment, I also went with a vnet
solution. Worked for me. Thank you.
@stblassitude What did you have to modify to make it work? Specifically where is the
vnet
option needed? In the jail Poudriere is in or in Poudriere-created jails?
It is in the configuration for the poudriere jail. I did this.
For /etc/rc.conf:
cloned_interfaces="bridge0"
ifconfig_bridge0="addm ix0 up"
Then for the jail configuration:
$ sudo iocage set vnet=on pkg01
$ sudo iocage set ip4_addr="vnet0|10.55.0.29/24" pkg01
$ sudo iocage set ip6_addr="vnet0|[redacted]:23/64" pkg01
I'll have a blog post in the next week or so.
Just wanted to follow up with an update I made while running 12.1-RELEASE-p7 after reading this:
"It also means that when a jailed daemon attaches to a port on the “loopback address,” it’s actually attaching to the jail’s external IP address."
Lucas, Michael W. FreeBSD Mastery: Jails (IT Mastery Book 15)
Setting LOIP4 in poudriere.conf to the jail's ip4_addr allowed ruby to compile.
In my case:
# iocage get ip4_addr poudriere
lo1|192.168.10.100
# grep LOIP4 /usr/local/etc/poudriere.conf
LOIP4=192.168.10.100
Most helpful comment
I've managed to get a working setup again, by using VIMAGE and a vnet jail. If the "classic" shared IP setup can be made to work, I would like to hear how.