Pecan: Mis-match between input prior data and output ensemble traits / runs when running with DVM-DOS-TEM PFT CMT04-Salix

Its very hard to tell right now but it seems this may only be related to this specific PFT? But not really sure, I have messed around with other modes by PFT combos and haven't seen this yet. Also could be specific to the tem model registration in pEcAn/bety

Here is a snippet of one of my pecan.CONFIGS.xmls

?xml version="1.0"?>
<pecan>
 <outdir>/data/sserbin/Modeling/dvmdostem/pecan_runs/run.56/</outdir>
 <database>
  <bety>
   <user>bety</user>
   <password>bety</password>
   <host>localhost</host>
   <dbname>bety</dbname>
   <driver>PostgreSQL</driver>
   <write>FALSE</write>
  </bety>
  <dbfiles>/data/home/sserbin/.pecan/dbfiles</dbfiles>
 </database>
 <pfts>
  <pft>
   <name>CMT04-Salix</name>
   <constants>
    <num>1</num>
   </constants>
   <posteriorid>2000001342</posteriorid>
   <outdir>/data/sserbin/Modeling/dvmdostem/pecan_runs/run.56//pft/CMT04-Salix</outdir>
  </pft>
 </pfts>

So I added some print statements to get.parameter.samples

### Identify PFTs in the input settings.xml file
get.parameter.samples <- function(settings, 
                                  posterior.files = rep(NA, length(settings$pfts)), 
                                  ens.sample.method = "uniform") {

  ## added by sps
  PEcAn.logger::logger.debug(paste0("get.param.samples.R posterior file: ", posterior.files))
  PEcAn.logger::logger.debug(paste0("priors: ", prior.distns))
  # added by sps


  pfts      <- settings$pfts
  num.pfts  <- length(settings$pfts)
  pft.names <- list()
  outdirs   <- list()

  for (i.pft in seq_along(pfts)) {
    pft.names[i.pft] <- settings$pfts[[i.pft]]$name

    ### If no PFT(s) are specified insert NULL to warn user
    if (length(pft.names) == 0) {
      pft.names[1] <- "NULL"
    }

    ### Get output directory info
    outdirs[i.pft] <- settings$pfts[[i.pft]]$outdir
    PEcAn.logger::logger.info(paste0("PFT Output Directories: ", outdirs)) # added by sps

  }  ### End of for loop to extract pft names

  PEcAn.logger::logger.info("Selected PFT(s): ", pft.names)

  ## Generate empty list arrays for output.
  trait.samples <- sa.samples <- ensemble.samples <- env.samples <- runs.samples <- list()

  ## Load PFT priors and posteriors
  for (i in seq_along(pft.names)) {
    #rm(prior.distns, post.distns, trait.mcmc) # sps
    ## Load posteriors
    if (!is.na(posterior.files[i])) {
      # Load specified file
      load(posterior.files[i])
      if (!exists("prior.distns") & exists("post.distns")) {
        prior.distns <- post.distns
      }
    } else {
      # Default to most recent posterior in the workflow, or the prior if there is none
      fname <- file.path(outdirs[i], "post.distns.Rdata")
      PEcAn.logger::logger.debug(paste0("posterior fname: ",fname))  # added by sps
      if (file.exists(fname)) {
        load(fname)
        prior.distns <- post.distns
      } else {
        load(file.path(outdirs[i], "prior.distns.Rdata"))
      }
    }

    ## added by sps
    PEcAn.logger::logger.debug(paste0("get.param.samples.R posterior file: ", posterior.files))
    PEcAn.logger::logger.debug(paste0("priors: ", prior.distns))
    # added by sps

    ### Load trait mcmc data (if exists)
    if ("trait.mcmc.Rdata" %in% dir(unlist(outdirs[i]))) {
      ma.results <- TRUE
      PEcAn.logger::logger.debug("*** Trait MCMC samples found ***") # added by sps
      load(file.path(outdirs[i], "trait.mcmc.Rdata"))
    } else {
      ma.results <- FALSE
      PEcAn.logger::logger.debug("*** Trait MCMC samples not found ***") # added by sps
    }

    pft.name <- unlist(pft.names[i])

Log output

Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
2017-10-10 16:54:40 WARN   [PEcAn.logger::logger.warn] :
   Multiple Valid Files found on host machine. Returning last updated
   record.
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
2017-10-10 16:54:40 DEBUG  [PEcAn.logger::logger.debug] :
   get.param.samples.R posterior file:
   /data/home/sserbin/.pecan/dbfiles/posterior/2000001348/post.distns.MA.Rdata
2017-10-10 16:54:40 DEBUG  [PEcAn.logger::logger.debug] :
   priors: c("gamma", "beta", "gamma", "gamma", "beta", "lnorm", "weibull",
   "beta", "weibull", "beta", "weibull", "norm", "weibull", "weibull",
   "gamma") priors: c(1.47, 2.63, 2.9, 6.53, 20.3, 0.811, 1.67, 3.61, 3.63,
   2, 2.66, 10, 2.06, 3.55, 90.9) priors: c(0.0578, 6.52, 0.63, 1.49, 76.1,
   0.843, 0.657, 0.433, 3.81, 4, 6.29, 1.02, 19, 24.7, 1580) priors: c(0,
   0, 40, 17, 30, 0, 66, 0, 4, 0, 35, 0, 125, 97, 56)
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
2017-10-10 16:54:40 INFO   [PEcAn.logger::logger.info] :
   PFT Output Directories:
   /data/sserbin/Modeling/dvmdostem/pecan_runs/run.60//pft/CMT04-Salix
2017-10-10 16:54:40 INFO   [PEcAn.logger::logger.info] :
   Selected PFT(s): CMT04-Salix
2017-10-10 16:54:40 DEBUG  [PEcAn.logger::logger.debug] :
   get.param.samples.R posterior file:
   /data/home/sserbin/.pecan/dbfiles/posterior/2000001348/post.distns.MA.Rdata
2017-10-10 16:54:40 DEBUG  [PEcAn.logger::logger.debug] :
   priors: c("gamma", "beta", "gamma", "gamma", "beta", "lnorm", "weibull",
   "beta", "weibull", "beta", "weibull", "norm", "weibull", "weibull",
   "gamma") priors: c(1.47, 2.63, 2.9, 6.53, 20.3, 0.811, 1.67, 3.61, 3.63,
   2, 2.66, 10, 2.06, 3.55, 90.9) priors: c(0.0578, 6.52, 0.63, 1.49, 76.1,
   0.843, 0.657, 0.433, 3.81, 4, 6.29, 1.02, 19, 24.7, 1580) priors: c(0,
   0, 40, 17, 30, 0, 66, 0, 4, 0, 35, 0, 125, 97, 56)
2017-10-10 16:54:40 DEBUG  [PEcAn.logger::logger.debug] :
   *** Trait MCMC samples found ***
2017-10-10 16:54:40 INFO   [PEcAn.logger::logger.info] :
   PFT CMT04-Salix has MCMC samples for: SLA
2017-10-10 16:54:40 INFO   [PEcAn.logger::logger.info] :
   PFT CMT04-Salix will use prior distributions for: mort2
   growth_resp_factor leaf_turnover_rate leaf_width nonlocal_dispersal
   fineroot2leaf root_turnover_rate seedling_mortality stomatal_slope
   r_fract root_respiration_rate Vm_low_temp Vcmax quantum_efficiency
2017-10-10 16:54:40 INFO   [PEcAn.logger::logger.info] :
   using 5004 samples per trait
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
Warning: '$.SafeList' is deprecated.

What is weird is that the first set of logger statements show that the run already has posterior.files and prior.distns set before it even gets to parsing priors/posteriors in the for loop? So I still cant isolate where the priors are coming from and how they are then getting used in that function? Based on the code it looks like it should be using the info from the PFT folder but they seem like they are coming in from elsewhere?

It sure seems that it should be runnign this bit to get the correct posterior info but for whatever reason it is no

    } else {
      # Default to most recent posterior in the workflow, or the prior if there is none
      fname <- file.path(outdirs[i], "post.distns.Rdata")
      PEcAn.logger::logger.debug(paste0("posterior fname: ",fname))  # added by sps
      if (file.exists(fname)) {
        load(fname)
        prior.distns <- post.distns
      } else {
        load(file.path(outdirs[i], "prior.distns.Rdata"))
      }
    }

I know that it isn't running because the logger output isn't there....

So instead of running that part, it is using some other prior.distns from somewhere else that it seems to be grabbing from who knows where???

OK, very very weird and I am lost with this. I created a NEW DVM-DOS-TEM PFT and I am still getting the same issue

Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
2017-10-13 08:28:17 INFO   [PEcAn.logger::logger.info] : 
   Selected PFT(s): CMT04-Betula 
Warning in rm(prior.distns, post.distns, trait.mcmc) :
  object 'prior.distns' not found
Warning in rm(prior.distns, post.distns, trait.mcmc) :
  object 'post.distns' not found
Warning in rm(prior.distns, post.distns, trait.mcmc) :
  object 'trait.mcmc' not found
2017-10-13 08:28:17 INFO   [PEcAn.logger::logger.info] : 
   PFT CMT04-Betula has MCMC samples for: SLA 
2017-10-13 08:28:17 INFO   [PEcAn.logger::logger.info] : 
   PFT CMT04-Betula will use prior distributions for: mort2 
   growth_resp_factor leaf_turnover_rate leaf_width nonlocal_dispersal 
   fineroot2leaf root_turnover_rate seedling_mortality stomatal_slope 
   r_fract root_respiration_rate Vm_low_temp Vcmax quantum_efficiency 
2017-10-13 08:28:17 INFO   [PEcAn.logger::logger.info] : 
   using 5004 samples per trait 
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")

So is there some reason this error could be model specific with respect to dvm-dos-tem? Perhaps somethign is getting masked improperly from the NAMESPACE? I can't see how this is happening?

OK, something is up. Here is a SIPNET run

Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
2017-10-13 08:32:10 INFO   [PEcAn.logger::logger.info] : 
   Selected PFT(s): tundra.deciduous.NGEE_Arctic 
Warning in rm(prior.distns, post.distns, trait.mcmc) :
  object 'prior.distns' not found
Warning in rm(prior.distns, post.distns, trait.mcmc) :
  object 'post.distns' not found
Warning in rm(prior.distns, post.distns, trait.mcmc) :
  object 'trait.mcmc' not found
2017-10-13 08:32:10 INFO   [PEcAn.logger::logger.info] : 
   PFT tundra.deciduous.NGEE_Arctic has MCMC samples for: leafC Amax 
   Vm_low_temp leaf_respiration_rate_m2 SLA 
2017-10-13 08:32:10 INFO   [PEcAn.logger::logger.info] : 
   PFT tundra.deciduous.NGEE_Arctic will use prior distributions for: mort2 
   growth_resp_factor leaf_turnover_rate leaf_width nonlocal_dispersal 
   fineroot2leaf root_turnover_rate seedling_mortality stomatal_slope 
   r_fract root_respiration_rate Vcmax quantum_efficiency 
2017-10-13 08:32:10 INFO   [PEcAn.logger::logger.info] : 
   using 3004 samples per trait 
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
2017-10-13 08:32:10 INFO   [PEcAn.logger::logger.info] : 
   No sampling method supplied, defaulting to uniform random sampling 
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
2017-10-13 08:32:10 INFO   [PEcAn.logger::logger.info] : 
   Using uniform random sampling 
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")

Clearly Vcmax and mort2 are an error. Where are these priors coming from!

Also, here is earlier output from my workflow.log before the meta.analysis

Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
2017-10-13 08:28:10 INFO   [PEcAn.logger::logger.info] : 
   --------------------------------------------------------- 
2017-10-13 08:28:10 INFO   [PEcAn.logger::logger.info] : SLA 
Auto-disconnecting PostgreSQLConnection
2017-10-13 08:28:13 INFO   [PEcAn.logger::logger.info] : Median SLA : 12.525 
2017-10-13 08:28:13 INFO   [PEcAn.logger::logger.info] : 
   --------------------------------------------------------- 
2017-10-13 08:28:13 INFO   [PEcAn.logger::logger.info] : 
   Summary of Prior distributions for: CMT04-Betula 
2017-10-13 08:28:13 INFO   [PEcAn.logger::logger.info] : 
   distn parama paramb n 
2017-10-13 08:28:13 INFO   [FUN] : 
   cuticular_cond lnorm 8.400 0.900 0 
2017-10-13 08:28:13 INFO   [FUN] : 
   extinction_coefficient_diffuse gamma 5.000 10.000 NA 
2017-10-13 08:28:13 INFO   [FUN] : 
   gcmax unif 0.001 0.005 NA 
2017-10-13 08:28:13 INFO   [FUN] : 
   SLA norm 10.000 5.000 NA 
2017-10-13 08:28:13 INFO   [FUN] : 
   frprod_perc_20 unif 0.100 99.000 NA 
2017-10-13 08:28:13 INFO   [FUN] : 
   frprod_perc_10 unif 0.100 99.000 NA 
2017-10-13 08:28:13 INFO   [PEcAn.logger::logger.info] : 
   number of observations per trait for CMT04-Betula 
2017-10-13 08:28:13 INFO   [PEcAn.logger::logger.info] : 
   34 observations of SLA 
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")

You can see the correct traits show up there, so the issue comes after meta-analysis and before running write.configs

OK, and then sometimes it will work fine, like below

Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
2017-10-13 08:48:05 INFO   [PEcAn.logger::logger.info] :
   PFT Output Directories:
   /data/sserbin/Modeling/dvmdostem/pecan_runs/run.61//pft/CMT04-Salix
2017-10-13 08:48:05 INFO   [PEcAn.logger::logger.info] :
   Selected PFT(s): CMT04-Salix
2017-10-13 08:48:05 DEBUG  [PEcAn.logger::logger.debug] :
   posterior fname:
   /data/sserbin/Modeling/dvmdostem/pecan_runs/run.61//pft/CMT04-Salix/post.distns.Rdata
2017-10-13 08:48:05 DEBUG  [PEcAn.logger::logger.debug] :
   get.param.samples.R posterior file: NA
2017-10-13 08:48:05 DEBUG  [PEcAn.logger::logger.debug] :
   priors: c("weibull", "unif", "unif", "norm", "unif", "unif", "weibull")
   priors: c(1.12, 0.01, 0.001, 10.9152064936782, 0.1, 0.1, 4) priors:
   c(54531, 0.3, 0.005, 0.180754050808388, 99, 99, 0.4) priors: c(29, NA,
   NA, NA, NA, NA, NA)
2017-10-13 08:48:05 DEBUG  [PEcAn.logger::logger.debug] :
   *** Trait MCMC samples found ***
2017-10-13 08:48:05 INFO   [PEcAn.logger::logger.info] :
   PFT CMT04-Salix has MCMC samples for: SLA
2017-10-13 08:48:05 INFO   [PEcAn.logger::logger.info] :
   PFT CMT04-Salix will use prior distributions for: cuticular_cond
   SW_albedo gcmax frprod_perc_20 frprod_perc_10
   extinction_coefficient_diffuse
2017-10-13 08:48:05 INFO   [PEcAn.logger::logger.info] :
   using 5004 samples per trait
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
Warning: '$.SafeList' is deprecated.

I really think there is something about order here....

Note in the log output above get.param.samples.R posterior file: NA and it worked fine, but further up that print statement shows a path to a file, a file with the correct info in it, but then it uses the wrong priors?

Yup, and if I run the EXACT SAME workflow and settings in pecan.xml, i.e. just source the workflow again I get different results!!!

See help("Deprecated")
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
2017-10-13 08:56:35 WARN   [PEcAn.logger::logger.warn] :
   Multiple Valid Files found on host machine. Returning last updated
   record.
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
2017-10-13 08:56:36 DEBUG  [PEcAn.logger::logger.debug] :
   get.param.samples.R posterior file:
   /data/home/sserbin/.pecan/dbfiles/posterior/2000001353/post.distns.MA.Rdata
2017-10-13 08:56:36 DEBUG  [PEcAn.logger::logger.debug] :
   priors: c("gamma", "beta", "gamma", "gamma", "beta", "lnorm", "weibull",
   "beta", "weibull", "beta", "weibull", "norm", "weibull", "weibull",
   "gamma") priors: c(1.47, 2.63, 2.9, 6.53, 20.3, 0.811, 1.67, 3.61, 3.63,
   2, 2.66, 10, 2.06, 3.55, 90.9) priors: c(0.0578, 6.52, 0.63, 1.49, 76.1,
   0.843, 0.657, 0.433, 3.81, 4, 6.29, 1.02, 19, 24.7, 1580) priors: c(0,
   0, 40, 17, 30, 0, 66, 0, 4, 0, 35, 0, 125, 97, 56)
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
2017-10-13 08:56:36 INFO   [PEcAn.logger::logger.info] :
   PFT Output Directories:
   /data/sserbin/Modeling/dvmdostem/pecan_runs/run.61//pft/CMT04-Salix
2017-10-13 08:56:36 INFO   [PEcAn.logger::logger.info] :
   Selected PFT(s): CMT04-Salix
2017-10-13 08:56:36 DEBUG  [PEcAn.logger::logger.debug] :

You can see that this time that print statement is not NA and I get the wrong traits in write.configs!!

So when it doesn't work correctly, it points to the CORRECT posterior file BUT somehow gets priors for traits that do not exist IN THAT FILE or in the PFT!?

So i dont know if this is helpful but @tobeycarman who is also running into this problem just accidentally started a workflow (sourced it) from the wrong starting directory so it did not find his PEcAn.xml. But the error message is interesting

> 
> 
> # ----------------------------------------------------------------------
> # PEcAn Workflow .... [TRUNCATED] 

> if (is.na(args[1])){
+   settings <- PEcAn.settings::read.settings("pecan.xml")
+ } else {
+   settings.file = args[1]
+   settings <- PEcAn.setting .... [TRUNCATED] 
2017-10-13 08:05:51 SEVERE [PEcAn.logger::logger.severe] : 
   Could not find a pecan.xml file 
Warning in file(file, ifelse(append, "a", "w")) :
  cannot open file '/home/dlebauer/out/ebifarm/post//STATUS': No such file or directory
Error in file(file, ifelse(append, "a", "w")) : 
  cannot open the connection
Error during wrapup: cannot open the connection
>

Is it possible that, somehow, there is a posterior in the database from an old run by @dlebauer, looks like an ED2 run, that we are somehow getting defaulted to? Or perhaps somewhere in an R package. I searched around without luck. I realize this is a whacky long shot but i figured I would ask in case it was something that would spark an idea

Because I can't see how his error would show up like that....unless it is something to do with the VM.

Here are outputs from @tobeycarman also showing the inconsistency where it wont then will then wont work again. But also he is getting the same incorrect parameters

output-1.txt
output-0.txt

I added TRUE in my meta-analysis tags, even though i a not certain that is a valid tag. However doing that did cause it to work properly for a single run

See help("Deprecated")
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
2017-10-13 09:57:01 WARN   [PEcAn.logger::logger.warn] :
   Multiple Valid Files found on host machine. Returning last updated
   record.
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
2017-10-13 09:57:01 DEBUG  [PEcAn.logger::logger.debug] :
   get.param.samples.R posterior file: NA
2017-10-13 09:57:01 DEBUG  [PEcAn.logger::logger.debug] :
   priors: c("gamma", "beta", "gamma", "gamma", "beta", "lnorm", "weibull",
   "beta", "weibull", "beta", "weibull", "norm", "weibull", "weibull",
   "gamma") priors: c(1.47, 2.63, 2.9, 6.53, 20.3, 0.811, 1.67, 3.61, 3.63,
   2, 2.66, 10, 2.06, 3.55, 90.9) priors: c(0.0578, 6.52, 0.63, 1.49, 76.1,
   0.843, 0.657, 0.433, 3.81, 4, 6.29, 1.02, 19, 24.7, 1580) priors: c(0,
   0, 40, 17, 30, 0, 66, 0, 4, 0, 35, 0, 125, 97, 56)
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.
See help("Deprecated")
2017-10-13 09:57:01 INFO   [PEcAn.logger::logger.info] :
   PFT Output Directories:
   /data/sserbin/Modeling/dvmdostem/pecan_runs/run.64//pft/CMT04-Salix
2017-10-13 09:57:01 INFO   [PEcAn.logger::logger.info] :
   Selected PFT(s): CMT04-Salix
2017-10-13 09:57:01 DEBUG  [PEcAn.logger::logger.debug] :
   posterior fname:
   /data/sserbin/Modeling/dvmdostem/pecan_runs/run.64//pft/CMT04-Salix/post.distns.Rdata
2017-10-13 09:57:01 DEBUG  [PEcAn.logger::logger.debug] :
   get.param.samples.R posterior file: NA
2017-10-13 09:57:01 DEBUG  [PEcAn.logger::logger.debug] :
   priors: c("weibull", "unif", "unif", "norm", "unif", "unif", "weibull")
   priors: c(1.12, 0.01, 0.001, 10.9152064936782, 0.1, 0.1, 4) priors:
   c(54531, 0.3, 0.005, 0.180754050808388, 99, 99, 0.4) priors: c(29, NA,
   NA, NA, NA, NA, NA)
2017-10-13 09:57:01 DEBUG  [PEcAn.logger::logger.debug] :
   *** Trait MCMC samples found ***
2017-10-13 09:57:01 INFO   [PEcAn.logger::logger.info] :
   PFT CMT04-Salix has MCMC samples for: SLA
2017-10-13 09:57:01 INFO   [PEcAn.logger::logger.info] :
   PFT CMT04-Salix will use prior distributions for: cuticular_cond
   SW_albedo gcmax frprod_perc_20 frprod_perc_10
   extinction_coefficient_diffuse
2017-10-13 09:57:01 INFO   [PEcAn.logger::logger.info] :
   using 5004 samples per trait
Warning: '$.SafeList' is deprecated.
Use 'PEcAn.settings::$.SafeList' instead.

So it seems if something causes pecan to pickup a change it works properly but otherwise it doesnt. And again get.param.samples.R posterior file: NA when it works

Doing that in the web did not have the same result so this is pretty inconsistent.

Adding the TRUE tag did not work for me.

@serbinsh could you maybe run the workflow line-by-line up until here
The "Multiple Valid Files found on host machine. Returning last updated
record. " warning could be a clue, dbfile.check can return a list of unwanted files sometimes

not sure that's relevant to your problem, but this was causing me problems on PDA, which is why I added return.all option to the function

I'm having the same problem with DALEC
My most recent run 1000008435

Produced a config file with

-mort2 19.5518192434762 
-growth_resp_factor 0.269134877386544 
-t5 0.00115746296987216 
-leaf_width 4.21460770704145 
-nonlocal_dispersal 0.208363236071724 
-fineroot2leaf 2.28414274764454 -t7 0.00143050364071698 
-seedling_mortality 0.95094009313231 
-stomatal_slope 3.44801807715082
-r_fract 0.31488851365167 
-root_respiration_rate 5.48891170784511 
-Vm_low_temp 9.99793554087408 
-SLA 0.0329788044168264 
-Vcmax 22.4703903038444 
-quantum_efficiency 0.0574557888428778

Which is definitely wrong variables.

Tried to Duplicate Betsy's run on the release VM and the DALEC config file was

The run succeeded, so clearly somehow the latest code on develop is defaulting to certain traits.

@tonygardella I agree it is a develop branch issue. My hunch is unintended consequence of moving functions and NAMESPACE around but can't yet figure out why

Tracked this down to write.configs step. Specifically at the get.parameter.samples step it's messing up finding the correct prior and/or posterior data and then defaulting to those ed specific traits.

@tonygardella Can we see the path/location where it is grabbing those priors?

I believe this is the offending file: https://github.com/PecanProject/pecan/blob/develop/base/utils/data/post.distns.RData

In get.parameter.samples this line removes the object of prior.distns such that when it gets to this line it can assign the posterior.distns to prior.distns, but in this case the loaded prior.distns.RData file still is loaded as prior.distns because the PEcAn.utils package is loaded. The existence of that object makes go on to use that package file as a default.

The reason this happens on our machines and not the VM is because that part of the code doesn't get triggered unless posterior files exist, which they don't on the VM.

When running it line by line I kept trying to delete prior.distns from my environment but it couldn't find the object, but then when you run the code it somehow had a prior.distns object, which led me to that file that is part of the utils package.

Is the file used for some test somewhere? https://github.com/PecanProject/pecan/blob/develop/base/utils/data/post.distns.RData

A quick search didn't show it used anywhere.

Aaaand this explains why it showed up when it did -- the in-package post.distns wasn't automatically visible in the workflow until I switched utils to lazy-loading in #1684. Mea culpa!