Poudriere: Try to fetch dependencies instead of building them locally

Created on 31 May 2015  ·  21Comments  ·  Source: freebsd/poudriere

Testing ports before committing changes is expensive. So, is building overlay of a few ports with non-default options.

  • /head is a fast moving target, ports -u often renders many out-of-date
  • Tier1 support consists of 4 * 2 = 8 platform combinations
  • redports.org (CI) is still not available after it went down a year ago
  • repo layout doesn't support NO_ARCH, NO_OSREL, NO_OPSYS yet
  • some ports may require a lot of memory/time/diskactivity to build

Fortunately, pkg.freebsd.org already provides ready-to-use packages. Why not take advantage? Caveat: only useful when local ports tree and remote snapshot at the time don't diverge much.

Performance

Most helpful comment

It's not yet implemented in Poudriere but I'm scheduled it for after the next release.

All 21 comments

Yes this is a wanted feature. I've called it 'package seeding'. The pkg.freebsd.org build script actually does this itself so we can build on different systems between each build. It's been on the TODOLIST forever but not yet implemented in Poudriere.

Also recently started to actively use poudriere to build packages for themselves and would like a similar function for the -f pkg-list.txt if part of a package with the changed options, then build them locally, and with default options fetch upstream repo.

Is this already implemented in a branch, I'd like to help to test it.

It's not yet implemented in Poudriere but I'm scheduled it for after the next release.

Hi, @bdrewery any update on this patch? Really need your feedback on how I can refine this patch. Any hint appreciated. Thanks a lot.

Sorry I'm been quite busy lately. I'll try to look in the next few weeks.

any new info? thanks!

This would be very nice feature to have. I just updated to 3.3.99.20190311, but didn't see an option for pkg seeding. Is there news on this feature?

In the mean time, I am using some hackish way to do this: https://github.com/lwhsu/freebsd-ports-libreoffice/blob/master/porttest.sh

Well, try to poudriere testport which needs both GCC 9.x and Clang 8.0 to build....
took about a day last time I needed to test that.

@bdrewery any hope to see this feature soon enough?

Not likely as my focus is elsewhere, such as building less in incremental build and distributed support.

Any update coming for this issue? Is there anything we can do to help?

Sorry for another "me too". I do like to build some ports that I'm using, as I've selected special flags on them. Things like Mesa et al. need a specific LLVM version though, and, well, I really don't want or need to build that myself. So it would be great if I could tell poudriere for some select ports to never build them, but fetch them instead.

Otherwise, my run ends up like so:

[freebsd:12:x86:64-default] [2020-06-22_09h45m06s] [parallel_build:] Queued: 1616 Built: 97   Failed: 0    Skipped: 0    Ignored: 0    Tobuild: 1519  Time: 02:21:47
        [01]: devel/llvm80              | llvm80-8.0.1_3            build           (01:50:32 / 01:51:56)
        [02]: devel/llvm90              | llvm90-9.0.1_1            build           (01:50:59 / 01:52:03)
        [03]: lang/gcc9                 | gcc9-9.3.0_1              build           (01:51:28 / 01:51:54)

I'd like to avoid that pointless churn. Thanks!

@uqs I hear you. I also think that there is another way to solve that particular problem, which is getting official packages built using the flags that you want. Could you share details on what flags you change?

Another me too here. Namely, I need the dns/unbound port with python, but I don't need to have all underlying dependencies built.

+1

+1

Either this feature or some documentation on how to use hooks to to achieve the same would be great!

797 will cover this

797 is merged but there are a lot of pitfalls that make this not as useful as it seems. For example, llvm and all the other big stuff still builds. #822 will probably be needed to fix that.

Was this page helpful?
0 / 5 - 0 ratings