Pecan: Clean up scripts directory

Oh good, I am confused! I am actually wrestling with this right now, trying to install pecan on a server I am working on. I just cloned a fresh copy of pecan and have SHA:82d217e checked out. I started trying the various install scripts in the pecan/scripts directory with the following results:

• Rscript install_deps.R: fails because it can't find package purrrs
• Rscript dependencies.R: fails because it can't find package (I am not sure what the package was but it was not purrrs
• Rscript generate_dependencies.R: fails because it can't find package purrrs
• Rscript install.dependencies.R: runs for a long time installing lots and lots of stuff until eventually failing with this:

ERROR: dependency ‘htmlTable’ is not available for package ‘Hmisc’
* removing ‘/data/home/tcarman/R/library/Hmisc’
ERROR: dependency ‘rgl’ is not available for package ‘Maeswrap’
* removing ‘/data/home/tcarman/R/library/Maeswrap’
ERROR: dependencies ‘usethis’, ‘DT’, ‘memoise’, ‘rcmdcheck’ are not available for package ‘devtools’
* removing ‘/data/home/tcarman/R/library/devtools’

The downloaded source packages are in
  ‘/tmp/Rtmpi2tei7/downloaded_packages’
There were 16 warnings (use warnings() to see them)
Error in loadNamespace(name) : there is no package called ‘devtools’
Calls: :: ... tryCatch -> tryCatchList -> tryCatchOne -> <Anonymous>
Execution halted

So what is the preferred way to use the install scripts in the scripts directory?

I think you can ignore most of those scripts and just use run make

Unfortunately make fails looking for devtools: Error in loadNamespace(name) : there is no package called ‘devtools’

Yeah, devtools is one of a small number of supporting packages that do need to be installed manually for the make to run. Would be good to fix/generalize that, but no one's gotten to it.

@tobeycarman Make should attempt to install devtools if needed at the beginning of installation, but it seems to be easy to confuse. Does make clean && make install help?

Well unfortunately simply make clean && make didn't work, but I did figure it out. I know the process is somewhat system/machine specific, but I here are the steps I followed. This may not be the best place to record this info, but here it is - maybe it will help someone in the future!

Deleted ~/R/library, logged out, logged in, made sure R path still set to my personal library.

Enabled modules:

$ module load gcc/5.4.0  gfortran/5.4.0  json-c/main  perl/5.22.1  git/2.10.0-rc0  git-lfs/1.3.1  python/3.6.2  udunits/2.2.25  proj/5.2.0  libtiff/4.0.8  geotiff/1.4.2  libkml/devel  geos/3.6.3  gdal/2.4.2  libxml2/2.9.7  jags/4.3.0  maui/3.3.1  redland/1.0.17 ImageMagick/7.0.8-19  hdf5/1.8.19-gcc540  netcdf/4.4.1.1-gnu540

Tried installing devtools, but it complained about a bunch of byte compiled
dependencied being built for the wrong internals.

Then try running update thinking maybe this will fix the dependencies for devtools:

> update.packages(checkBuilt=TRUE, ask=FALSE)

Then try installing devtools again:

> install.packages("devtools")

It worked! Next need rgdal and sf which I know from S. Serbin don't work well
when installed with automatic dependency resolution. And we discoverd a bug
with the most recent rgdal, so I grab a recent version.

$ cd ~/downloads && wget https://cran.r-project.org/src/contrib/Archive/rgdal/rgdal_1.4-4.tar.gz

Launch R with a few env vars set and install rgdal from source:

$ CXX="g++ -std=gnu++11" R

> install.packages('~/downloads/rgdal_1.4-4.tar.gz', repos = NULL, type="source", configure.args=c('--with-proj-include=/data/software/proj/5.2.0/include','--with-proj-lib=/data/software/proj/5.2.0/lib'))

Logout, and log back in (remember to load modules!!). This addresses an error finding proj_api.h. Now install
sf:

$ CXX="g++ -std=gnu++11" R

> install.packages('sf', configure.args=c('--with-proj-include=/data/software/proj/5.2.0/include','--with-proj-lib=/data/software/proj/5.2.0/lib', '--with-proj=/data/software/proj/5.2.0/'))

Now change into my cloned pecan directory and try make. This fails with a number of errors about finding headers and libs for postgres and redland. After some digging around to find the right paths, I finally can install those by setting env variables
before launching R:

$ INCLUDE_DIR="/data/software/postgresql/9.3.5/include" LIB_DIR="/data/software/postgresql/9.3.5/lib" R
> install.packages("RPostgres")

And

$ PKG_CONFIG_PATH="/data/software/redland/1.0.17/lib/pkgconfig:/data/software/raptor/2.0.15/lib/pkgconfig:/data/software/rasqal/0.9.33/lib/pkgconfig" R

> install.packages("redland")

Then try pecan's make again, and (after well over an hour!) it seems to work!
I started with an empty ~/R/library and at the end this is what I have:

$ ls ~/R/library
abind
ape
askpass
assertthat
backports
base64enc
BayesianTools
BH
bibtex
binaryLogic
BioCro
bit
bit64
bitops
blob
boot
brew
bridgesampling
Brobdingnag
broom
callr
car
carData
CDM
cellranger
class
classInt
cli
clipr
clisymbols
cluster
coda
codetools
colorspace
commonmark
covr
crayon
crosstalk
crul
curl
dataone
datapack
data.table
DBI
dbplyr
desc
devtools
DHARMa
digest
doParallel
dotCall64
dplyr
DT
e1071
ellipse
ellipsis
emulator
evaluate
expm
fansi
fastmap
fauxpas
fields
forcats
foreach
foreign
fs
furrr
future
gap
generics
geometry
geonames
getPass
ggmap
ggplot2
gh
git2r
glmmTMB
globals
glue
gmm
gridExtra
gtable
hash
haven
hdf5r
here
highr
hms
hoardr
htmltools
htmlwidgets
httpcode
httpuv
httr
IDPmisc
igraph
ini
isdparser
iterators
jpeg
jsonlite
KernSmooth
knitr
labeling
later
lazyeval
lifecycle
linkages
linprog
listenv
lme4
lmtest
lpSolve
lubridate
Maeswrap
magic
magrittr
manipulateWidget
maps
maptools
markdown
MASS
Matrix
MatrixModels
mclust
mcmc
MCMCpack
memoise
mgcv
microbenchmark
mime
miniUI
minpack.lm
minqa
mlegp
mockery
modelr
MODISTools
msm
munsell
mvnfast
mvtnorm
ncdf4
ncdf4.helpers
nimble
nlme
nloptr
nneo
numDeriv
openssl
openxlsx
parsedate
pbapply
pbkrtest
pbv
PCICt
PEcAn.all
PEcAn.allometry
PEcAn.assim.batch
PEcAn.assim.sequential
PEcAn.BASGRA
PEcAn.benchmark
PEcAn.BIOCRO
PEcAn.CLM45
PEcAn.DALEC
PEcAn.data.atmosphere
PEcAn.data.hydrology
PEcAn.data.land
PEcAn.data.remote
PEcAn.DB
PEcAn.dvmdostem
PEcAn.ED2
PEcAn.emulator
PEcAn.FATES
PEcAn.GDAY
PEcAn.JULES
PEcAn.LINKAGES
PEcAn.logger
PEcAn.LPJGUESS
PEcAn.MA
PEcAn.MAAT
PEcAn.MAESPA
PEcAn.ModelName
PEcAn.photosynthesis
PEcAn.PRELES
PEcAn.priors
PEcAn.qaqc
PEcAn.remote
PEcAnRTM
PEcAn.settings
PEcAn.SIPNET
PEcAn.uncertainty
PEcAn.utils
PEcAn.visualization
PEcAn.workflow
pillar
pkgbuild
pkgconfig
pkgload
plogr
plotrix
plyr
png
polycor
praise
prettyunits
processx
progress
promises
proxy
ps
purrr
pwr
qrnn
quantreg
R6
randtoolbox
rappdirs
raster
rcmdcheck
RColorBrewer
Rcpp
RcppArmadillo
RcppEigen
RcppProgress
rcrossref
RCurl
readr
readxl
REddyProc
redland
rematch
rematch2
remotes
reprex
reshape
reshape2
reticulate
rex
rgdal
rgl
RgoogleMaps
rio
rjags
rjson
rlang
rmarkdown
rngWELL
rnoaa
roxygen2
rpart
RPostgres
RPostgreSQL
Rpreles
rprojroot
RSQLite
rstudioapi
rversions
rvest
sandwich
scales
selectr
sessioninfo
sf
sfsmisc
shiny
SimilarityMeasures
sirt
solartime
sourcetools
sp
spam
spaMM
SparseM
stringi
stringr
survival
sys
TAM
testthat
tibble
tictoc
tidyr
tidyselect
tidyverse
tinytex
TMB
tmvtnorm
triebeard
truncnorm
udunits2
units
urltools
usethis
utf8
uuid
vctrs
viridisLite
webshot
whisker
withr
xfun
XML
xml2
xopen
xtable
xts
yaml
zeallot
zip
zoo

@tobeycarman This is a glorious effort for a process that really ought to be easier than this. Thanks for sticking with it!

Part of @tobeycarman challenge comes from working on modex, which is a HPC with shared storage thus everything is compiled by hand in non-standard locations, e.g. gdal, proj, etc. But also yeah it would be good for us to clean up the automated prerequisite tools to make running make install bulletproof

@infotroph hey, thanks! @serbinsh, while bulletproof make install would be awesome, I can see how it might be a tall ask, especially for HPC systems. For the record I fully support cleaning up the scripts directory by removing files or marking them as "deprecated in favor of..."