Thanks for the report.

Item 4 from NEWS for 1.10.4-1 :

Attempted improvement on Mac with Intel’s OpenMP when package ‘parallel’ forks after data.table has performed in parallel with OpenMP. That OpenMP implementation appears to leave threads running after the parallel region has finished. If this fix still doesn’t work, call setDTthreads(1) immediately after library(data.table) which has been reported to fix the problem.

In dev I improved that same item but neglected to backport that news items change to the CRAN patch. From dev NEWS file :

Attempted improvement on Mac-only when the parallel package is used too (which forks), #2137. Intel's OpenMP implementation appears to leave threads running after the OpenMP parallel region (inside data.table) has finished unlike GNU libgomp. So, if and when parallel's fork is invoked by the user after data.table has run in parallel already, instability occurs. The problem only occurs with Mac package binaries from CRAN because they are built by CRAN with Intel's OpenMP library. No known problems on Windows or Linux and no known problems on any platform when parallel is not used. If this Mac-only fix still doesn't work, call setDTthreads(1) immediately after library(data.table) which has been reported to fix the problem by putting data.table into single threaded mode earlier.

However, in your case, you haven't even used data.table before forking. So that is a new aspect that hasn't been reported before. Is your reproducible example strictly correct in that those 3 lines are literally all that occurs in a fresh R session? i.e. no usage of data.table between loading it and calling mclapply? You could try calling setDTthreads(1) immediately after library(data.table) anyway, but I doubt it would help if there really is no usage of data.table before the mclapply.

Have you installing the CRAN mac binary, or from source? It appears to be critical on MacOS, with respect to this mclapply issue which OpenMP library you use. Are you able to establish whether it's Intel's (libomp) or GNU's (libgomp) OpenMP implementation that's being used on machine?

Can you compile from source using the MacOS instructions and does that help? Maybe that links to the proper OpenMP library locally for you, or something like that. Some people have not followed the instructions precisely enough, so they have to be followed precisely on MacOS.

Does test.data.table() work for you? Test 1705 tests mclapply and it is passing on CRAN. I assume CRAN tests the Mac binaries on MacOS, but it's possible I suppose that it tests a different build.

Item 1 from NEWS from 1.10.4-2 I doubt is related, but I'll mention it for completeness :

OpenMP on MacOS is now supported by CRAN and included in CRAN’s package binaries for Mac. But installing v1.10.4-1 from source on MacOS failed when OpenMP was not enabled at compile time, #2409. Thanks to Liz Macfie and @fupangpangpang for reporting. The startup message when OpenMP is not enabled has been updated.

Here are related issues :
https://github.com/Rdatatable/data.table/issues/2413
https://github.com/Rdatatable/data.table/issues/2409
https://github.com/Rdatatable/data.table/issues/2406
https://github.com/Rdatatable/data.table/issues/2137

I'm basically struggling with MacOS. If there is anyone listening from Apple and/or Intel who can support their products, that would be great. I struggle to see how it's a data.table issue really. I've tried everything I can think of to work around it and document it in NEWS. Help and suggestions much appreciated.

Yes, the simple script with only

library(parallel)
library(data.table)
mclapply(1:2, sqrt)

This is on the Bioconductor build machines. I believe this is a simple binary installation, but will confirm. We have

$ ls /Library/Frameworks/R.framework/Resources/lib
libR.dylib      libRlapack.dylib.dSYM   libquadmath.0.dylib
libR.dylib.dSYM     libc++.1.dylib      libreadline.5.2.dylib
libRblas.0.dylib    libc++abi.1.dylib   libreadline.dylib
libRblas.dylib      libgcc_s.1.dylib    libunwind.1.dylib
libRblas.dylib.dSYM libgfortran.3.dylib
libRlapack.dylib    libomp.dylib

and

$ otool -L  /Library/Frameworks/R.framework/Resources/bin/exec/R 
/Library/Frameworks/R.framework/Resources/bin/exec/R:
    /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libR.dylib (compatibility version 3.4.0, current version 3.4.2)
    /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRblas.dylib (compatibility version 0.0.0, current version 0.0.0)
    /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libomp.dylib (compatibility version 5.0.0, current version 5.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)

So presumably linking to libomp.

test.data.table() says

Running test id 1705     Assertion failure at kmp_runtime.cpp(6480): __kmp_thread_pool == __null.
Assertion failure at kmp_runtime.cpp(6480): __kmp_thread_pool == __null.
OMP: Error #13: Assertion failure at kmp_runtime.cpp(6480).
OMP: Hint: Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see http://www.intel.com/software/products/support/.
OMP: Error #13: Assertion failure at kmp_runtime.cpp(6480).
OMP: Hint: Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see http://www.intel.com/software/products/support/.
Assertion failure at kmp_runtime.cpp(6480): __kmp_thread_pool == __null.
OMP: Error #13: Assertion failure at kmp_runtime.cpp(6480).
OMP: Hint: Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see http://www.intel.com/software/products/support/.
Assertion failure at kmp_runtime.cpp(6480): __kmp_thread_pool == __null.
OMP: Error #13: Assertion failure at kmp_runtime.cpp(6480).
OMP: Hint: Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see http://www.intel.com/software/products/support/.
Running test id 1750     Test 1751 not run. If required call library(nanotime) first.

endian==little, sizeof(long double)==16, sizeof(pointer)==8

All 5936 tests in inst/tests/tests.Rraw completed ok in 00:01:45 on Sat Oct 14 21:25:30 2017

Thanks for this. Yes those __kmp_* calls are, I believe, specific to the Intel OpenMP Runtime and I've seen them reported before. Test 1705 is specifically testing that. It utilizes 2 OpenMP threads within data.table itself followed by calling parallel::mclapply. The __kmp_ errors are because, iiuc, Intel's implementation leaves the threads running even after all parallel regions have finished, so the subsequent fork is unstable. In this case it looks like it didn't crash, but that's just luck. The fact that you're seeing that output with 1.10.4-2 means my attempted workaround did not work. There is a pre-fork callback in which it now calls omp_set_num_threads(1) in the hopes that the Intel library would clear up its running threads before the fork. But it looks like it doesn't. So, we are left with what has been reported to work, which is to call setDTthreads(1) immediately after library(data.table) to stop Intel's OpenMP creating the threads in the first place.

On the hang with the mere 3 lines though, is it possible Intel's OpenMP library creates its threads upon being linked or initialized? If so, that might explain that. Otherwise, I'm at a loss on that.

Further, I don't know how the OpenMP abilities of clang and gcc relate to the 3 OpenMP libraries: libomp, libiomp and libgomp. You seem to have libomp. But I think CRAN compiles with libiomp? Maybe that's to do with it. Also, one person has reported (here) that "changing llvm to gcc" stopped the crashes on MacOS, but they were compiling from source.

I just came across the source code for the Intel OpenMP runtime. I don't know what this project is because I thought Intel's library was closed source. But maybe it's possible to compile against it and debug it. https://www.openmprtl.org/ Probably easier just to compile locally first and see if that works, or use GNU library.

Test 1705 is passing on CRAN so that's perhaps hopeful, by finding out what's different between the CRAN machine(s) and your machine. Test 1705 is actually passing for you and on CRAN, but libomp is outputting assertion errors to the console then which aren't getting picking up by the test. I wonder if it's possible to see the CRAN logs somehow to see if those __kmp messages are in CRAN's logs too. If the messages aren't present in CRAN's logs, then it might be worth exploring the difference between CRAN machine and your machine.

Maybe this is needed with the CRAN package binary: http://macappstore.org/libiomp/
Or perhaps libiomp is compiled statically into the package binary by CRAN? I noticed that the 'Binary packages' section of ?install.packages is quite complicated for MacOS and there are several binary options there for the type= argument.

Intel's library is printing :

OMP: Error #13: Assertion failure at kmp_runtime.cpp(6480)
OMP: Hint: Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see http://www.intel.com/software/products/support/.

Could someone with the problem on MacOS file an issue with Intel please?

Hi, I originally reported the issue with the BiocParallel repository as I encountered the issue while using functions from that package. They've narrowed the issue to mclapply being called while data.table is loaded.

I followed the directions here to install the latest version of data.table on my MacOS, but still encountered the issue.

Interestingly, a colleague had the latest version of data.table installed on his Mac and did not have any issues. However, he did not have OpenMP installed on his system.

I was able to fix the problem by rolling back to data.table 1.10.2.

@asenabouth Can you raise a support request with Intel please (see my comment above)? It appears to be a problem with their library. Alternatively, perhaps someone from Apple can help. Are you able to get their support at all? I'm not convinced that installing from source does not solve it. Can you show the commands and output please. Is anyone else unable to use parallel::mclapply after data.table 1.10.4 has been loaded, even after compiling from source and avoiding the CRAN binary?

I will file a support request with Intel.

As for installing from source, here is the installation and subsequent test:
```{r}
[Previously saved workspace restored]

remove.packages("data.table")
Removing package from ‘/Users/a.senabouth/Library/R/3.4/library’
(as ‘lib’ is unspecified)
install.packages("data.table", type = "source",

• repos = "http://Rdatatable.github.io/data.table")
  Installing package into ‘/Users/a.senabouth/Library/R/3.4/library’
  (as ‘lib’ is unspecified)
  trying URL 'http://Rdatatable.github.io/data.table/src/contrib/data.table_1.10.5.tar.gz'

Content type 'application/gzip' length 3312731 bytes (3.2 MB)

downloaded 3.2 MB

• installing source package ‘data.table’ ...
  * libs
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c assign.c -o assign.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c between.c -o between.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c bmerge.c -o bmerge.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c chmatch.c -o chmatch.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c dogroups.c -o dogroups.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fastmean.c -o fastmean.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fcast.c -o fcast.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fmelt.c -o fmelt.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c forder.c -o forder.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c frank.c -o frank.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fread.c -o fread.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c freadR.c -o freadR.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fsort.c -o fsort.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fwrite.c -o fwrite.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fwriteR.c -o fwriteR.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c gsumm.c -o gsumm.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c ijoin.c -o ijoin.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c init.c -o init.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c inrange.c -o inrange.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c openmp-utils.c -o openmp-utils.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c quickselect.c -o quickselect.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c rbindlist.c -o rbindlist.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c reorder.c -o reorder.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c shift.c -o shift.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c subset.c -o subset.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c transpose.c -o transpose.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c uniqlist.c -o uniqlist.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c vecseq.c -o vecseq.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c wrappers.c -o wrappers.o
  /usr/local/opt/llvm/bin/clang -fopenmp -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/opt/gettext/lib -L/usr/local/opt/llvm/lib -Wl,-rpath,/usr/local/opt/llvm/lib -o data.table.so assign.o between.o bmerge.o chmatch.o dogroups.o fastmean.o fcast.o fmelt.o forder.o frank.o fread.o freadR.o fsort.o fwrite.o fwriteR.o gsumm.o ijoin.o init.o inrange.o openmp-utils.o quickselect.o rbindlist.o reorder.o shift.o subset.o transpose.o uniqlist.o vecseq.o wrappers.o -fopenmp -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
  mv data.table.so datatable.so
  if [ "" != "Windows_NT" ] && [ uname -s = 'Darwin' ]; then install_name_tool -id datatable.so datatable.so; fi
  installing to /Users/a.senabouth/Library/R/3.4/library/data.table/libs
  * R
  * inst
  * byte-compile and prepare package for lazy loading
  * help
  ** installing help indices
  * building package indices
  * installing vignettes
  ** testing if installed package can be loaded
• DONE (data.table)

The downloaded source packages are in
‘/private/var/folders/zz/zyxvpxvq6csfxvn_n0001lp0000cxh/T/Rtmpl7TqS4/downloaded_packages’

R
Error: object 'R' not found
library(parallel)
library(data.table)
data.table 1.10.5 IN DEVELOPMENT built 2017-10-16 07:29:07 UTC; travis
The fastest way to learn (by data.table authors): https://www.datacamp.com/courses/data-analysis-the-data-table-way
Documentation: ?data.table, example(data.table) and browseVignettes("data.table")
Release notes, videos and slides: http://r-datatable.com
mclapply(1:2, sqrt)

* caught illegal operation *
address 0x7fffa9730ae4, cause 'illegal opcode'

Traceback:
1: mcfork()
2: FUN(X[[i]], ...)
3: lapply(seq_len(cores), inner.do)
4: mclapply(1:2, sqrt)

Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace

And here's the session info:
```{r}
R version 3.4.2 (2017-09-28)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Sierra 10.12.6

Matrix products: default
BLAS: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib

locale:
[1] en_AU.UTF-8/en_AU.UTF-8/en_AU.UTF-8/C/en_AU.UTF-8/en_AU.UTF-8

attached base packages:
[1] parallel  stats     graphics  grDevices utils     datasets  methods  
[8] base     

other attached packages:
[1] data.table_1.10.5

loaded via a namespace (and not attached):
[1] compiler_3.4.2

EDIT:
I got it to work by installing 1.10.4 from source.
```{r}

require(devtools)
Loading required package: devtools
install_version("data.table", version = "1.10.4", repos = "http://cran.us.r-project.org")
Downloading package from url: http://cran.us.r-project.org/src/contrib/Archive/data.table/data.table_1.10.4.tar.gz
Installing data.table
'/Library/Frameworks/R.framework/Resources/bin/R' --no-site-file --no-environ
--no-save --no-restore --quiet CMD INSTALL
'/private/var/folders/zz/zyxvpxvq6csfxvn_n0001lp0000cxh/T/RtmpXDSgR8/devtoolsdf05280c79d9/data.table'
--library='/Users/a.senabouth/Library/R/3.4/library' --install-tests

• installing source package ‘data.table’ ...
  * package ‘data.table’ successfully unpacked and MD5 sums checked
  * libs
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c assign.c -o assign.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c between.c -o between.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c bmerge.c -o bmerge.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c chmatch.c -o chmatch.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c dogroups.c -o dogroups.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fastmean.c -o fastmean.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fcast.c -o fcast.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fmelt.c -o fmelt.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c forder.c -o forder.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c frank.c -o frank.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fread.c -o fread.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fsort.c -o fsort.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fwrite.c -o fwrite.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c gsumm.c -o gsumm.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c ijoin.c -o ijoin.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c init.c -o init.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c inrange.c -o inrange.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c openmp-utils.c -o openmp-utils.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c quickselect.c -o quickselect.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c rbindlist.c -o rbindlist.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c reorder.c -o reorder.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c shift.c -o shift.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c subset.c -o subset.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c transpose.c -o transpose.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c uniqlist.c -o uniqlist.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c vecseq.c -o vecseq.o
  /usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -fopenmp -fPIC -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c wrappers.c -o wrappers.o
  /usr/local/opt/llvm/bin/clang -fopenmp -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/opt/gettext/lib -L/usr/local/opt/llvm/lib -Wl,-rpath,/usr/local/opt/llvm/lib -o data.table.so assign.o between.o bmerge.o chmatch.o dogroups.o fastmean.o fcast.o fmelt.o forder.o frank.o fread.o fsort.o fwrite.o gsumm.o ijoin.o init.o inrange.o openmp-utils.o quickselect.o rbindlist.o reorder.o shift.o subset.o transpose.o uniqlist.o vecseq.o wrappers.o -fopenmp -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
  mv data.table.so datatable.so
  if [ "" != "Windows_NT" ] && [ uname -s = 'Darwin' ]; then install_name_tool -id datatable.so datatable.so; fi
  installing to /Users/a.senabouth/Library/R/3.4/library/data.table/libs
  * R
  * inst
  * tests
  * byte-compile and prepare package for lazy loading
  * help
  ** installing help indices
  * building package indices
  * installing vignettes
  ** testing if installed package can be loaded
• DONE (data.table)
  > library(data.table)
  data.table 1.10.4
  The fastest way to learn (by data.table authors): https://www.datacamp.com/courses/data-analysis-the-data-table-way
  Documentation: ?data.table, example(data.table) and browseVignettes("data.table")
  Release notes, videos and slides: http://r-datatable.com
  > library(parallel)
  > mclapply(1:2, sqrt)
  [[1]]
  [1] 1

[[2]]
[1] 1.414214
```

I've posted the following https://software.intel.com/en-us/forums/intel-c-compiler/topic/746943

It's probably in the wrong place and I'll probably get flamed but hopefully it will get the ball rolling.

@mattdowle let me know if there are things you would like me to add/do

Wanted to confirm that I reproduced the same error as @asenabouth and that reinstall 1.10.4 from source fixed the problem.

sessionInfo:

R version 3.4.2 (2017-09-28)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS High Sierra 10.13

Matrix products: default
BLAS: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] parallel  stats     graphics  grDevices utils     datasets  base     

other attached packages:
[1] data.table_1.10.4

loaded via a namespace (and not attached):
[1] compiler_3.4.2 methods_3.4.2 

Just logging some details on my machine

library(parallel)
library(data.table)
mclapply(1:2, sqrt)

produces

 *** caught illegal operation ***
address 0x7fffb500eae4, cause 'illegal opcode'

with sessionInfo()

R version 3.4.2 (2017-09-28)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Sierra 10.12.6

Matrix products: default
BLAS: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib

locale:
[1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8

attached base packages:
[1] parallel  stats     graphics  utils     datasets  grDevices methods  
[8] base     

other attached packages:
[1] data.table_1.10.4-2 testthat_1.0.2      devtools_1.13.3    

loaded via a namespace (and not attached):
[1] compiler_3.4.2 magrittr_1.5   R6_2.2.2       withr_2.0.0    crayon_1.3.4  
[6] memoise_1.1.0  digest_0.6.12

Interestingly,

> remove.packages("data.table")
Removing package from ‘/Library/Frameworks/R.framework/Versions/3.4/Resources/library’
(as ‘lib’ is unspecified)
> install.packages("data.table", type = "source",
+ repos = "http://Rdatatable.github.io/data.table")

produces

trying URL 'http://Rdatatable.github.io/data.table/src/contrib/data.table_1.10.5.tar.gz'

Content type 'application/gzip' length 3312726 bytes (3.2 MB)
==================================================
downloaded 3.2 MB

* installing *source* package ‘data.table’ ...
** libs
clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/include  -fopenmp -fPIC  -Wall -g -O2  -c assign.c -o assign.o
clang: error: unsupported option '-fopenmp'
make: *** [assign.o] Error 1
ERROR: compilation failed for package ‘data.table’
* removing ‘/Library/Frameworks/R.framework/Versions/3.4/Resources/library/data.table’

The downloaded source packages are in
    ‘/private/var/folders/48/q6ltldjs251000_wvjrdy_vm0000gn/T/Rtmp6eZFXj/downloaded_packages’
Warning message:
In install.packages("data.table", type = "source", repos = "http://Rdatatable.github.io/data.table") :
  installation of package ‘data.table’ had non-zero exit status

I'm going to see if this is related

I also followed the instructions at https://github.com/Rdatatable/data.table/wiki/Installation and the problem still occurs....

EDIT:
The original problem with mclapply and the illegal operation (not the compiler error unsupported option '-fopenmp' - that was fixed by following the instructions)

Thanks everybody. It's interesting that 1.10.4 appears to work ok but 1.10.4-2 does not. It appears to crash as soon as the fork is called. The attempt I mentioned in NEWS was in that area.

First, what didn't change. Upon loading data.table, init.c calls this in openmp-utils.c :

void avoid_openmp_hang_within_fork() {
  // Called once on loading data.table from init.c
  #ifdef _OPENMP
    pthread_atfork(&when_fork, NULL, NULL);
  #endif
}

so when parallel::mclapply calls its fork(), the when_fork function runs. Which in 1.10.4-2 looks like this :

void when_fork() {    # v 1.10.4-2
  // attempted workaround for Intel's OpenMP implementation which leaves threads
  // running after parallel region; these crash when forked.
  #ifdef _OPENMP
    omp_set_num_threads(1);
  #endif

  DTthreads = 1;
}

before (in 1.10.4), it didn't used to call omp_set_num_threads(1), and it was just setting that static variable DTthreads :

void when_fork() {    # v 1.10.4
  DTthreads = 1;
}

All I can think is that call to omp_set_num_threads(1) is crashing it. But that's the simplest function you can call in OpenMP. Perhaps it has not been bound properly to the library, somehow?

1. For the people that can reproduce the crash when calling parallel::mclapply, does simple fwrite() work ok for you in a fresh session without loading package parallel? Does it use many CPUs as it should? You will need a large DT to test otherwise it just uses 1 thread for small data. If it crashes upon any usage of OpenMP then we have a linking problem. Does anyone know what these flags in the link line mean and might they be something to do with it on MacOS? : -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress

2. I could try removing that call to omp_set_num_threads(1) again in dev and someone with the problem now with 1.10.5 could try dev again after I change it?

3. The strange thing is that tests are passing on CRAN on MacOS with no crash. Test 1705 tests calling parallel::mclapply and all the tests run on CRAN using 2 threads. What's the difference between the CRAN machine and the machines that people have the problem with?

@joethorley That's odd. You get unsupported option '-fopenmp' even after running the MacOS instructions here. Can anyone else suggest how to progress then? I'm at a loss.

@joethorley Do you have gettext installed? Turns out I didn't. I was able to compile the package correctly on my system once that was done.

@mattdowle no the unsupported option '-fopenmp' error disappeared after I followed the instructions here but the error

 *** caught illegal operation ***
address 0x7fffb500eae4, cause 'illegal opcode'

occurred.

My Makevars is

CC=/usr/local/opt/llvm/bin/clang -fopenmp
CXX=/usr/local/opt/llvm/bin/clang++
# -O3 should be faster than -O2 (default) level optimisation ..
CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe
LDFLAGS=-L/usr/local/opt/gettext/lib -L/usr/local/opt/llvm/lib -Wl,-rpath,/usr/local/opt/llvm/lib
CPPFLAGS=-I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include

My sessionInfo()

R version 3.4.2 (2017-09-28)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Sierra 10.12.6

Matrix products: default
BLAS: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib

locale:
[1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8

attached base packages:
[1] stats     graphics  utils     datasets  grDevices methods   base     

other attached packages:
[1] testthat_1.0.2  devtools_1.13.3

loaded via a namespace (and not attached):
[1] compiler_3.4.2 magrittr_1.5   R6_2.2.2       withr_2.0.0    crayon_1.3.4  
[6] memoise_1.1.0  digest_0.6.12 

and my R session

> remove.packages("data.table")
Removing package from ‘/Library/Frameworks/R.framework/Versions/3.4/Resources/library’
(as ‘lib’ is unspecified)
> install.packages("data.table", type = "source", repos = "http://Rdatatable.github.io/data.table")
trying URL 'http://Rdatatable.github.io/data.table/src/contrib/data.table_1.10.5.tar.gz'
Content type 'application/gzip' length 3312684 bytes (3.2 MB)
==================================================
downloaded 3.2 MB

* installing *source* package ‘data.table’ ...
** libs
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c assign.c -o assign.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c between.c -o between.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c bmerge.c -o bmerge.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c chmatch.c -o chmatch.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c dogroups.c -o dogroups.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fastmean.c -o fastmean.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fcast.c -o fcast.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fmelt.c -o fmelt.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c forder.c -o forder.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c frank.c -o frank.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fread.c -o fread.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c freadR.c -o freadR.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fsort.c -o fsort.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fwrite.c -o fwrite.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fwriteR.c -o fwriteR.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c gsumm.c -o gsumm.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c ijoin.c -o ijoin.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c init.c -o init.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c inrange.c -o inrange.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c openmp-utils.c -o openmp-utils.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c quickselect.c -o quickselect.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c rbindlist.c -o rbindlist.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c reorder.c -o reorder.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c shift.c -o shift.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c subset.c -o subset.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c transpose.c -o transpose.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c uniqlist.c -o uniqlist.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c vecseq.c -o vecseq.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c wrappers.c -o wrappers.o
/usr/local/opt/llvm/bin/clang -fopenmp -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/opt/gettext/lib -L/usr/local/opt/llvm/lib -Wl,-rpath,/usr/local/opt/llvm/lib -o data.table.so assign.o between.o bmerge.o chmatch.o dogroups.o fastmean.o fcast.o fmelt.o forder.o frank.o fread.o freadR.o fsort.o fwrite.o fwriteR.o gsumm.o ijoin.o init.o inrange.o openmp-utils.o quickselect.o rbindlist.o reorder.o shift.o subset.o transpose.o uniqlist.o vecseq.o wrappers.o -fopenmp -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
mv data.table.so datatable.so
if [ "" != "Windows_NT" ] && [ `uname -s` = 'Darwin' ]; then install_name_tool -id datatable.so datatable.so; fi
installing to /Library/Frameworks/R.framework/Versions/3.4/Resources/library/data.table/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (data.table)

The downloaded source packages are in
    ‘/private/var/folders/48/q6ltldjs251000_wvjrdy_vm0000gn/T/RtmpWku3yc/downloaded_packages’
> library(parallel)
> library(data.table)
data.table 1.10.5 IN DEVELOPMENT built 2017-10-18 23:12:32 UTC; travis
  The fastest way to learn (by data.table authors): https://www.datacamp.com/courses/data-analysis-the-data-table-way
  Documentation: ?data.table, example(data.table) and browseVignettes("data.table")
  Release notes, videos and slides: http://r-datatable.com
> mclapply(1:2, sqrt)

 *** caught illegal operation ***
address 0x7fffb500eae4, cause 'illegal opcode'

@mattdowle

1. fwrite() appears to work on my machine

> mtcars <- rbind(mtcars, mtcars)
> nrow(mtcars)                                       
[1] 16777216
> fwrite(mtcars, "mtcars.csv")
Written 45.1% of 16777216 rows in 2 secs using 8 threads. anyBufferGrown=no; max
Written 69.5% of 16777216 rows in 3 secs using 8 threads. anyBufferGrown=no; max
Written 93.9% of 16777216 rows in 4 secs using 8 threads. anyBufferGrown=no; max                                                                         

@asenabouth I do have gettext installed and I was able to compile the package correctly once I follow the instructions. My post was a little unclear so I have edited.

@mattdowle

1. If you remove that call to omp_set_num_threads(1) in dev I'll try dev again.

Installing 1.10.4.2 from CRAN, either source or binary, results in the mclapply hang. Installing github.com/Rdatatable/data.table@master from source still produced the original mclapply hang.

Installing this modified version of the current master

$ git diff
diff --git a/src/openmp-utils.c b/src/openmp-utils.c
index d6e3b70..bf104fc 100644
--- a/src/openmp-utils.c
+++ b/src/openmp-utils.c
@@ -78,7 +78,7 @@ void when_fork() {
 void avoid_openmp_hang_within_fork() {
   // Called once on loading data.table from init.c
 #ifdef _OPENMP
-  pthread_atfork(&when_fork, NULL, NULL);
+//  pthread_atfork(&when_fork, NULL, NULL);
 #endif
 }
 ```

or this version

$ git diff
diff --git a/src/openmp-utils.c b/src/openmp-utils.c
index d6e3b70..8c0236d 100644
--- a/src/openmp-utils.c
+++ b/src/openmp-utils.c
@@ -62,7 +62,7 @@ void when_fork() {
// attempted workaround for Intel's OpenMP implementation which leaves threads running after
// parallel region; these crash when forked.
#ifdef _OPENMP

• omp_set_num_threads(1);
  +// omp_set_num_threads(1);
  #endif

// GNU OpenMP seems ok with just setting DTthreads to 1 which limits the next parallel region

clears the hang for me.

The R we use (installed as a binary from CRAN) is the 3.4.2 release

R.version
_
platform x86_64-apple-darwin15.6.0
arch x86_64
os darwin15.6.0
system x86_64, darwin15.6.0
status
major 3
minor 4.2
year 2017
month 09
day 28
svn rev 73368
language R
version.string R version 3.4.2 (2017-09-28)
nickname Short Summer
```

Setting the environment variable OMP_NUM_THREADS=2 did not change the outcome.

I can confirm commenting out omp_set_num_threads(1); in when_fork() fixes the problem on my machine as well

> mclapply(1:2, sqrt)
[[1]]
[1] 1

[[2]]
[1] 1.414214

Thanks everyone. Made the fix in dev and is passing. @joethorley Would you or someone else confirm that latest 1.10.5 is ok now. I'll start preparing 1.10.4-3 to go to CRAN.

@mattdowle

I can confirm it is fixed.

$ R

R version 3.4.2 (2017-09-28) -- "Short Summer"
Copyright (C) 2017 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin15.6.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> remove.packages("data.table")
Removing package from ‘/Library/Frameworks/R.framework/Versions/3.4/Resources/library’
(as ‘lib’ is unspecified)
> install.packages("data.table", type = "source", repos = "http://Rdatatable.github.io/data.table")
trying URL 'http://Rdatatable.github.io/data.table/src/contrib/data.table_1.10.5.tar.gz'
Content type 'application/gzip' length 3313284 bytes (3.2 MB)
==================================================
downloaded 3.2 MB

* installing *source* package ‘data.table’ ...
** libs
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c assign.c -o assign.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c between.c -o between.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c bmerge.c -o bmerge.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c chmatch.c -o chmatch.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c dogroups.c -o dogroups.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fastmean.c -o fastmean.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fcast.c -o fcast.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fmelt.c -o fmelt.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c forder.c -o forder.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c frank.c -o frank.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fread.c -o fread.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c freadR.c -o freadR.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fsort.c -o fsort.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fwrite.c -o fwrite.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c fwriteR.c -o fwriteR.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c gsumm.c -o gsumm.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c ijoin.c -o ijoin.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c init.c -o init.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c inrange.c -o inrange.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c openmp-utils.c -o openmp-utils.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c quickselect.c -o quickselect.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c rbindlist.c -o rbindlist.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c reorder.c -o reorder.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c shift.c -o shift.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c subset.c -o subset.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c transpose.c -o transpose.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c uniqlist.c -o uniqlist.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c vecseq.c -o vecseq.o
/usr/local/opt/llvm/bin/clang -fopenmp -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include  -fopenmp -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c wrappers.c -o wrappers.o
/usr/local/opt/llvm/bin/clang -fopenmp -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/opt/gettext/lib -L/usr/local/opt/llvm/lib -Wl,-rpath,/usr/local/opt/llvm/lib -o data.table.so assign.o between.o bmerge.o chmatch.o dogroups.o fastmean.o fcast.o fmelt.o forder.o frank.o fread.o freadR.o fsort.o fwrite.o fwriteR.o gsumm.o ijoin.o init.o inrange.o openmp-utils.o quickselect.o rbindlist.o reorder.o shift.o subset.o transpose.o uniqlist.o vecseq.o wrappers.o -fopenmp -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
mv data.table.so datatable.so
if [ "" != "Windows_NT" ] && [ `uname -s` = 'Darwin' ]; then install_name_tool -id datatable.so datatable.so; fi
installing to /Library/Frameworks/R.framework/Versions/3.4/Resources/library/data.table/libs
** R
** inst
** byte-compile and prepare package for lazy loading
session** help
In*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (data.table)

The downloaded source packages are in
    ‘/private/var/folders/48/q6ltldjs251000_wvjrdy_vm0000gn/T/RtmpmAMLnD/downloaded_packages’
> library(parallel)
> library(data.table)
data.table 1.10.5 IN DEVELOPMENT built 2017-10-19 19:32:41 UTC; travis
  The fastest way to learn (by data.table authors): https://www.datacamp.com/courses/data-analysis-the-data-table-way
  Documentation: ?data.table, example(data.table) and browseVignettes("data.table")
  Release notes, videos and slides: http://r-datatable.com
> sessionInfo()
R version 3.4.2 (2017-09-28)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Sierra 10.12.6

Matrix products: default
BLAS: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib

locale:
[1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8

attached base packages:
[1] parallel  stats     graphics  utils     datasets  grDevices methods  
[8] base     

other attached packages:
[1] data.table_1.10.5 testthat_1.0.2    devtools_1.13.3  

loaded via a namespace (and not attached):
[1] compiler_3.4.2 magrittr_1.5   R6_2.2.2       tools_3.4.2    withr_2.0.0   
[6] crayon_1.3.4   memoise_1.1.0  digest_0.6.12 
> mclapply(1:2, sqrt)
[[1]]
[1] 1

[[2]]
[1] 1.414214

Many thanks @joethorley! Submitted to 1.10.4-3 to CRAN and all being well, it will take a few days for the magic of CRAN to run its course.

Changes in v1.10.4-3 (on CRAN 20 Oct 2017)

1. Fixed crash/hang on MacOS when parallel::mclapply is used and data.table is merely loaded, #2418. Oddly, all tests including test 1705 (which tests mclapply with data.table) passed fine on CRAN. It appears to be some versions of MacOS or some versions of libraries on MacOS, perhaps. Many thanks to Martin Morgan for reporting and confirming this fix works. Thanks also to asenabouth, Joe Thorley and Danton Noriega for testing, debugging and confirming that automatic parallelism inside data.table (such as fwrite) works well even on these MacOS installations. See also news items below for 1.10.4-1 and 1.10.4-2.

Closing for now. Please reopen or open a new issue if there's a problem with 1.10.4-3.

@mattdowle: it looks like v1.10.4-2 is still the version being distributed by CRAN. Is v1.10.4-3 still waiting in the CRAN submission queue, or did something else go wrong?

(thank you for getting a workaround out for this issue so quickly; it unfortunately appears to be affecting a large number of users on macOS)

@kevinushey Yes 1.10.4-3 was waiting in the queue (in the 'recheck' status which I don't know what that means, for 6 days). I sent an email to CRAN and it's now released to CRAN.

maybe I am making something wrong but I still have the issue on my mac.

> sessionInfo()
R version 3.4.1 (2017-06-30)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Sierra 10.12.6

Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib

locale:
[1] fr_FR.UTF-8/fr_FR.UTF-8/fr_FR.UTF-8/C/fr_FR.UTF-8/fr_FR.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] httr_1.3.1      compiler_3.4.1  R6_2.2.2        tools_3.4.1     withr_2.0.0     curl_3.0       
 [7] yaml_2.1.14     memoise_1.1.0   knitr_1.17      git2r_0.19.0    digest_0.6.12   devtools_1.13.3

When installing the cran version from source, I get:

> install.packages("data.table", type = "source", repos = "https://cran.rstudio.com")
essai de l'URL 'https://cran.rstudio.com/src/contrib/data.table_1.10.4-3.tar.gz'
Content type 'application/x-gzip' length 3071833 bytes (2.9 MB)
==================================================
downloaded 2.9 MB

* installing *source* package ‘data.table’ ...
** package ‘data.table’ correctement décompressé et sommes MD5 vérifiées
** libs
clang -I/Library/Frameworks/R.framework/Resources/include -DNDEBUG   -I/usr/local/include  -fopenmp -fPIC  -Wall -g -O2  -c assign.c -o assign.o
clang: error: unsupported option '-fopenmp'
make: *** [assign.o] Error 1
ERROR: compilation failed for package ‘data.table’
* removing ‘/Library/Frameworks/R.framework/Versions/3.4/Resources/library/data.table’
* restoring previous ‘/Library/Frameworks/R.framework/Versions/3.4/Resources/library/data.table’

Les packages source téléchargés sont dans
    ‘/private/var/folders/51/6jygptvs3bb4njv0t6x7br900000gn/T/Rtmp8NJ0di/downloaded_packages’
Warning message:
In install.packages("data.table", type = "source", repos = "https://cran.rstudio.com") :
  l'installation du package ‘data.table’ a eu un statut de sortie non nul

I can provide more details if necessary.

@davidgohel: Please see this guide for getting the R LLVM toolchain installed locally. http://thecoatlessprofessor.com/programming/openmp-in-r-on-os-x/

Alternatively, I would recommend editing the file at R.home("etc/Makeconf") and just removing any instance of -fopenmp you see in there, so that compilation doesn't try to use OpenMP with the Apple system toolchain.

@kevinushey OK, I will have a try with the alternative solution. I think I messed something with the first solution.

Adding this to my ~/.R/Makevars file worked for compiling data.table without OpenMP on High Sierra:

# Suppress `-fopenmp` in R.home("etc/Makeconf")
# MAIN_LDFLAGS = -fopenmp
# SHLIB_OPENMP_CFLAGS = -fopenmp
# SHLIB_OPENMP_CXXFLAGS = -fopenmp
# SHLIB_OPENMP_FCFLAGS = -fopenmp
# SHLIB_OPENMP_FFLAGS = -fopenmp
MAIN_LDFLAGS =
SHLIB_OPENMP_CFLAGS =
SHLIB_OPENMP_CXXFLAGS =
SHLIB_OPENMP_FCFLAGS =
SHLIB_OPENMP_FFLAGS =

@mjsteinbaugh @kevinushey thanks, it worked

I tried editing the file at R.home("etc/Makeconf") but I am not allowed to edit it. I also don't seem to have a file called ~/.R/Makevars. Any further suggestions :) Thanks

@ValeriVoev The file at ~/.R/Makevars normally doesn't exist; you'll have to create it yourself. (That is, create a folder called .R in your home directory, and then create a Makevars file within that folder)

Thanks @kevinushey - I finally made it work, installed data.table 1.10.4-3 and R Studio doesn't crash anymore. I also installed clang - not sure whether that also helped or not.

I still see this error randomly

Assertion failure at kmp_runtime.cpp(6480): __kmp_thread_pool == __null.
OMP: Error #13: Assertion failure at kmp_runtime.cpp(6480).
OMP: Hint: Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see http://www.intel.com/software/products/support/.

And the case is really tricky.

1. data.table 1.10.4 and 1.10.4-3 are same for me.
2. I have a vignette with some parallel code that can run without problem when I run all chunks. However knit that vignette will have the error above.
3. If I put a simple parallel code mclapply(1:2, sqrt) in the beginning of the vignette, knitting the vignette no longer have the error!

This all seemed to be related to environment changes. Unfortunately I don't have a simple reproducible example now. My code is quite complex and involves a lot of other things.

Like dracodoc, have the (6480) catch randomly. macOS 10.13.3, R 3.4.3, data.table 1.10.4-3. Just plain R here and too complex code to reproduce.

Simple mclapply's at the start of the code work and sometimes make the rest of the code work as well. In my case, adding mclapply(1:2, sqrt) would not do the trick, but running a seemingly random number of trials does (now start the code with for (i in 1:10) mclapply(1:10000000, sqrt)).

EDIT: Alas, the execution does not seem to be stable, and though mcmapply starts, it hangs after some time.

I've just seen these further comments. Please either try dev 1.10.5 or wait for next CRAN release (hopefully soon). If still a problem after that, please raise a new issue.

I'm still seeing the same problem with dev 1.10.5. New issue created.