Zfs: Feature Request: Cached and paralleled DKMS

Sounds like you're trying to re-implement gentoo/funtoo (or similar source code based distributions) which already solved handling of system wide compile flags (and be it only the amount of gcc's running in parallel).

IMHO it's no good idea to interfere with system settings regarding the compiler. In case long build times are a problem for DKMS user... adding a howto ('speed up compilation') would be the reasonable approach.

You can already do what you are requesting by creating a DKMS config file. Look at the man page for dkms under _DKMS.CONF_ and _DKMS.CONF OVERRIDES_ for information on how to specify directives to override options specified in the dkms.conf which comes with DKMS package.

As an example, for RHEL/CentOS/Fedora and ZFS 0.7.13, if you look at the dkms.conf file which is provided with the RPMs (e.g., /var/lib/dkms/spl/0.7.13/source/dkms.conf and /var/lib/zfs/0.7.13/source/dkms.conf), you can see that the SPL and ZFS builds use only MAKE[0]="make". To override that, create the files /etc/dkms/spl.conf and /etc/dkms/zfs.conf. In the files, you can specify the MAKE command to use. To just enable a parallel build use:

MAKE[0]="make --jobs"

This will run all jobs in parallel and not limit the number which are run. A better option would be to limit it to the number of processors on the machine:

MAKE[0]="make --jobs $(nproc)"

However, none of the above should be necessary as modern versions of DKMS default to running make in parallel with the number of jobs set to the number of processors on the system. Look at /usr/sbin/dkms on your system. On mine, it has lines like these:

get_num_cpus()
{
   # use nproc(1) from coreutils 8.1-1+ if available, otherwise single job
   if [ -x /usr/bin/nproc ]; then
        nproc
   else
        echo "1"
  fi
}
...
do_build()
{
    ...
    local the_make_command="${make_command/#make/make -j$parallel_jobs KERNELRELEASE=$kernelver}"

    invoke_command "{ $the_make_command; } >> $dkms_tree/$module/$module_version/build/make.log 2>&1" "$the_make_command" background || \
        report_build_problem 10 $"Bad return status for module build on kernel: $kernelver ($arch)" \
        $"Consult $dkms_tree/$module/$module_version/build/make.log for more information."
...
parallel_jobs=${parallel_jobs:-$(get_num_cpus)}

# Make sure we're not passing -j0 to make; treat -j0 as just "-j"
[[ "$parallel_jobs" = 0 ]] && parallel_jobs=""

Since spl.dkms and zfs.dkms set MAKE[0]="make", dkms should already be running the make command in parallel.

Enabling ccache is documented in its man page. If you use the symlinks method, as long as /usr/local/bin is in your path before /usr/bin, ccache will be used automatically. If you wish to use the prefix command method, you should redefine CC in your local environment to something like ccache gcc.

I suspect that you'll find that neither of these changes make much difference. There aren't many files that have to be compiled and gcc is pretty fast (and DKMS should already be running make in parallel). My experience has been that the build step is not the one which takes a long time. The pre-build steps (e.g., configure) and post-build steps (e.g., dracut) are what take a long time on my machines.

I recommend profiling the build process on your machine to find out what is really taking a long time and focus on optimizing those parts.

On October 14, 2019 2:58 pm, Christopher Voltz wrote:

I suspect that you'll find that neither of these changes make much difference. There aren't many files that have to be compiled and gcc is pretty fast (and DKMS should already be running make in parallel). My experience has been that the build step is not the one which takes a long time. The pre-build steps (e.g., configure) and post-build steps (e.g., dracut) are what take a long time on my machines.

I recommend profiling the build process on your machine to find out what is really taking a long time and focus on optimizing those parts.

Also note that the biggest chunk of those pre-build steps is determining
what interfaces the kernel has, and that has just been (mostly)
parallelized in master. Our configure walltime dropped from ~2.5 minutes
to ~20 seconds. It was the biggest part of building the modules and
userspace. I suspect once that change makes it way to Debian, DKMS
module building will also be greatly sped up on systems with more than
say 2 or 4 cores.

You can already do what you are requesting by creating a DKMS config file. Look at the man page for dkms under _DKMS.CONF_ and _DKMS.CONF OVERRIDES_ for information on how to specify directives to override options specified in the dkms.conf which comes with DKMS package.

As an example, for RHEL/CentOS/Fedora and ZFS 0.7.13, if you look at the dkms.conf file which is provided with the RPMs (e.g., /var/lib/dkms/spl/0.7.13/source/dkms.conf and /var/lib/zfs/0.7.13/source/dkms.conf), you can see that the SPL and ZFS builds use only MAKE[0]="make". To override that, create the files /etc/dkms/spl.conf and /etc/dkms/zfs.conf. In the files, you can specify the MAKE command to use. To just enable a parallel build use:

MAKE[0]="make --jobs"

This will run all jobs in parallel and not limit the number which are run. A better option would be to limit it to the number of processors on the machine:

MAKE[0]="make --jobs $(nproc)"

However, none of the above should be necessary as modern versions of DKMS default to running make in parallel with the number of jobs set to the number of processors on the system. Look at /usr/sbin/dkms on your system. On mine, it has lines like these:

get_num_cpus()
{
   # use nproc(1) from coreutils 8.1-1+ if available, otherwise single job
   if [ -x /usr/bin/nproc ]; then
        nproc
   else
        echo "1"
  fi
}
...
do_build()
{
    ...
    local the_make_command="${make_command/#make/make -j$parallel_jobs KERNELRELEASE=$kernelver}"

    invoke_command "{ $the_make_command; } >> $dkms_tree/$module/$module_version/build/make.log 2>&1" "$the_make_command" background || \
        report_build_problem 10 $"Bad return status for module build on kernel: $kernelver ($arch)" \
        $"Consult $dkms_tree/$module/$module_version/build/make.log for more information."
...
parallel_jobs=${parallel_jobs:-$(get_num_cpus)}

# Make sure we're not passing -j0 to make; treat -j0 as just "-j"
[[ "$parallel_jobs" = 0 ]] && parallel_jobs=""

Since spl.dkms and zfs.dkms set MAKE[0]="make", dkms should already be running the make command in parallel.

Enabling ccache is documented in its man page. If you use the symlinks method, as long as /usr/local/bin is in your path before /usr/bin, ccache will be used automatically. If you wish to use the prefix command method, you should redefine CC in your local environment to something like ccache gcc.

I suspect that you'll find that neither of these changes make much difference. There aren't many files that have to be compiled and gcc is pretty fast (and DKMS should already be running make in parallel). My experience has been that the build step is not the one which takes a long time. The pre-build steps (e.g., configure) and post-build steps (e.g., dracut) are what take a long time on my machines.

I recommend profiling the build process on your machine to find out what is really taking a long time and focus on optimizing those parts.

@cvoltz Wow, I'm ashamed to have posted a feature request without digging deeper into what you've provided... you're right, those are both viable options that keep the ownership away from ZFS and even the package management side of things (which is where I thought this thread would go).

And yes, after going through the logs... you're correct... the config step is a huge laggard for me, much less on the compile side.

Anyways, I appreciate your response, as well as everyone elses... I'll close this up... sorry for the noise!

Cheers
Chris