Pecan: SIPNET bug fix and version control

I maintained a SIPNET svn repository at the University of Wisconsin through about 2010. I dug up some old emails from @jmzobitz from 2011 discussing the transfer of the svn repository to Augsburg, but I can't find details on that (like its URL), and I'm not sure if that was ever actively used.

If it would be helpful, I could probably dig up the old svn repo with history through 2010.

And:

• I would support moving SIPNET to GitHub

• No opinion on the bug mentioned here

I have an email from @jmzobitz from 2012 saying the svn is on Augsburg.

I'm also in favor of moving SIPNET to Github.

Hi all -
Thanks for starting this conversation.

The server to access sipnet is: http://turing.augsburg.edu/svn/sipnet, I
can look up your SVN password if you need it - just message me individually.

I support moving this to gitbhub, and here is some perspective:

• The last inquiry / etc I've had about the SIPNET repository was in 2016.
  I would get about 1 inquiry a year or so about access.
• Since that time my Department has moved to a brand new building, so I am
  unsure of the svn repository server is still hooked up.

We should move it to github - happy to host it wherever it might make sense
(my own github, with PEcAN project, etc).

Is there interest in moving this to other software platforms beyond C? (R?
Julia? Python?). I ask just to see what the future needs for SIPNET are so
we get the architecture / design of the github repo set up to support that
now.

Cheers,
John

John

John M. Zobitz, PhD
Associate Professor
Department of Mathematics, Statistics, and Computer Science
Environmental Studies Program
Augsburg University
2211 Riverside Avenue, CB 93
Minneapolis, MN 55454

Hagfors Center 187
P: (612) 330-1068
F: (612) 330-1393

On Mon, Jun 3, 2019 at 2:31 PM Michael Dietze notifications@github.com
wrote:

I have an email from @jmzobitz https://github.com/jmzobitz from 2012
saying the svn is on Augsburg.

I'm also in favor of moving SIPNET to Github.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/PecanProject/pecan/issues/2367?email_source=notifications&email_token=ACPZNTXXF7R3NBJJIMPJFBDPYVWPXA5CNFSM4HSOZP72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODW2OJRI#issuecomment-498394309,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACPZNTXPIZEED7OA344OFNLPYVWPXANCNFSM4HSOZP7Q
.

Thank you all for for your quick responses.

@jmzobitz thanks for the information. In PEcAn project, SIPNET is our go-to model for testing our modules from benchmarking to data assimilation. We often feel the need to point to a version-controlled SIPNET source code for collaboration, documentation and publication purposes. Occasionally, we fix a bug or add new features. SVN could also do, but I think the link above is not responding anymore. Overall, it would be great to have it on github.

As to where to host it, I would defer to your decision. In my opinion, it makes sense for you to host it. But if it would be easier for everyone, I think we can also host it under PEcAn Project. What would @mdietze and @robkooper think?

We have no problem with SIPNET being in C, we like it for being fast after all. But implementing SIPNET code in R or writing a wrapper might have certain benefits from our perspective. Not sure if there would be any interest for Julia and Python, but maybe @ashiklom has an opinion.

I'm indifferent between having the SIPNET code in John's Github vs creating a new SIPNET repo at https://github.com/pecanproject/

I agree with Istem that I don't think rewriting SIPNET into another language is a good use of time.

If I were to do anything to the SIPNET code it would be to clean it up and simplify it -- there's a bunch of hard-coded sensitivity analysis and MCMC code in there that we don't use because those are things PEcAn handles natively. I'd also consider switching the model to use netCDF inputs/outputs (in PEcAn standard) natively so we don't have to deal with ASCII <-> netCDF conversions (or perhaps having a settings flag to toggle between ASCII and netCDF as the former is definitely simpler for basic users). Those conversions often take us more compute time than the model run itself.

I disagree with Istem about having an R wrapper -- to me that's what write.configs.SIPNET in PEcAn already is and creating an additional wrapper would create ambiguity and redundancy.

Glad to hear support for moving SIPNET to GitHub! My personal vote would be to host it under the PEcAn organization, since that will make it easy to transfer ownership/maintenance responsibilities between individuals without changing the URL. But I'm open to any of the suggestions above.

I do feel strongly that SIPNET should be re-implemented in Haskell, or better yet, LOLCODE 🙊

But actually, I agree that a re-write isn't a great idea. That said, I do think there is some value in a thin R package wrapper (using Rcpp, or just R's built-in C interface). I don't think it should be anywhere near the top of anybody's priority list, but I do think the broader ecology community would benefit from having more models that could be quickly installed and run entirely from R. For instance:

install.packages("rsipnet")
library(rsipnet)
results <- run_sipnet("/path/to/sipnet.in", Vcmax = 28, ...)

A comparable example is the Hector model developed by my colleagues here at JGCRI -- that started out as a standalone C++ binary, but recently wrapping it in an R package (via Rcpp) has made it much easier to use, develop, and maintain.

but I do think the broader ecology community would benefit from having more models that could be quickly installed and run entirely from R

+1 to that, I recently wished that I could just run sipnet by simply installing an R-package for another project, instead I had to defer to VSEM which is implemented in the BayesianTools package.

I'd also consider switching the model to use netCDF inputs/outputs (in PEcAn standard) natively

+1 to that as well

Let's host it in the PEcAn project repository - that just makes good long
term sense.

As a side project I would be willing to re-write SIPNET to R - I believe
I started to do that in the past, then stopped. I see restarting that as a
good use of ongoing development my R functional programming skills - and I
know more of how R does some of that data management / processing via the
tidyverse.

If we agree to use netCDF as the data input standard, then I can write the
rsipnet package with that in mind.

My C coding skills are weak, so I can't re-write SIPNET to ingest netCDF
data directly.

John

John M. Zobitz, PhD
Associate Professor
Department of Mathematics, Statistics, and Computer Science
Environmental Studies Program
Augsburg University
2211 Riverside Avenue, CB 93
Minneapolis, MN 55454

Hagfors Center 187
P: (612) 330-1068
F: (612) 330-1393

On Tue, Jun 4, 2019 at 9:22 AM istfer notifications@github.com wrote:

but I do think the broader ecology community would benefit from having
more models that could be quickly installed and run entirely from R

+1 to that, I recently wished that I could just run sipnet by simply
installing an R-package for another project, instead I had to defer to
VSEM https://rdrr.io/cran/BayesianTools/man/VSEM.html which is
implemented in the BayesianTools package.

I'd also consider switching the model to use netCDF inputs/outputs (in
PEcAn standard) natively

+1 to that as well

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/PecanProject/pecan/issues/2367?email_source=notifications&email_token=ACPZNTXJ4QUARXVISWYHBCLPYZ3CLA5CNFSM4HSOZP72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODW4XL5A#issuecomment-498693620,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACPZNTT6LXOWGBY3WYJAFRDPYZ3CLANCNFSM4HSOZP7Q
.

John:

• Googling suggests that git svn clone should help with the SVN to GIT migration
• I just temporarily invited you to PEcAn-admin team so you can create and add the SIPNET repo. Once it's up let me know and I'll change permissions so that you don't have admin over all of PEcAn, just that repo.

Thanks @mdietze. I will check with colleagues at Augsburg to see if turing is hooked up anymore so we can do the clone.

I will let you know before I do any cloning and try it out on my github first - out of an abundance of not mucking up the PEcAn repository. :)

Thanks for getting the SIPNET code here. If you need help let me know and I can help with the conversion from SVN to GIT while preserving the logs. With a little magic we can even try to match the usernames to the appropriate emails if people have an account on GitHub so the commits will be associated with their GitHub account.

I'd also be happy to help with the conversion - or at least help review it - before it goes "live" (mainly to make sure history and authorship seem to be preserved properly). As @robkooper says, it could be great to maintain authorship... I think this can be done with an "authors" file argument to git svn clone.

Happy to help with code review, tests, too.

I talked with my CSC colleague - turns out turing went down a few years ago but I was failed to be notified. He's looking into if the files are recoverable, but we should think through an alternative option. For what it's worth, the svn SIPNET has not been modified since we started it.

@jmzobitz There's a good chance I have the old sipnet svn repo (as of 2010) saved on an external HD somewhere, so I can try to dig that up if you can't recover it on your end.

(A final fallback could be to just create a fresh repo without trying to maintain history. I do like preserving history when possible, though I can't remember how much of SIPNET's history actually existed in the svn repo: I know we weren't using version control for its very first development in the early 2000s; I can't remember when we moved it into svn.)

Folks

I have versions of SIPNET used for fluxcourse over the years and I'm also pretty confident i have the version used in various papers over the years after changes to add roots, microbes & also explicit error estimates on observations.

I don't have access right now as i'm in Ireland but I could dig a bit when i get back to Tucson for a few days at the end of June.
Cheers
Dave

Sent from my Verizon, Samsung Galaxy smartphone

-------- Original message --------
From: Bill Sacks notifications@github.com
Date: 6/4/19 9:57 PM (GMT+00:00)
To: PecanProject/pecan pecan@noreply.github.com
Cc: "Moore, David Joseph - (davidjpmoore)" davidjpmoore@email.arizona.edu, Mention mention@noreply.github.com
Subject: Re: [PecanProject/pecan] SIPNET bug fix and version control (#2367)

(A final fallback could be to just create a fresh repo without trying to maintain history. I do like preserving history when possible, though I can't remember how much of SIPNET's history actually existed in the svn repo: I know we weren't using version control for its very first development in the early 2000s; I can't remember when we moved it into svn.)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/PecanProject/pecan/issues/2367?email_source=notifications&email_token=ABO2T2BIVS52ORCPYUW36YLPY3JILA5CNFSM4HSOZP72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODW53HNY#issuecomment-498840503, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABO2T2GFCZXGTCDOB5RDARTPY3JILANCNFSM4HSOZP7Q.

I found a backup of the old repository (last trunk commit mid-2008, plus a branch from 2009 from @davidjpmoore ). I think it will be easiest if I do the initial git svn clone myself from this. Then I'll push it to a repo in my personal GitHub space for review, after which we can migrate it to PecanProject.

Thanks @billsacks!

Okay, I have done the git conversion and pushed a repo for review by @davidjpmoore @jmzobitz and anyone else who would like to review it before we migrate it to its final home in PecanProject:

https://github.com/billsacks/sipnet

I have preserved authorship information such that GitHub connects your contributions with your GitHub account. I have also done some massaging of the git history so that the existing feature branches show up properly. I have not done any of the cleanup that this repository desperately needs (like removing some started-but-never-finished projects that seem to have been done as file copies in the main branch): My goal at this point was to get a good starting point, where we are fairly confident that we won't need to further rewrite any of the pre-existing history once this "goes live".

@davidjpmoore @jmzobitz and others: can you please review to make sure that this reflects the final state that you expect to see? Also feel free to review for anything else you'd like (history, etc.)

@davidjpmoore - if you'd like, you can also review your two old feature branches here (moore and DoubleExponential) and say (1) whether these look right to you, and/or (2) whether you want these in the final repo or whether you'd prefer they be removed.

I should add: I have diffed the result against the svn repo backup that I had, so I'm pretty confident that the git version matches my svn version (both for master and the two feature branches). What I'm less confident about is whether the version I had was really the most up-to-date version: the last changes I had were:

• trunk: June 4, 2008

• DoubleExponential branch: Aug 20, 2007

• moore branch: Aug 1, 2009

@davidjpmoore I also found this email (April, 2011) when looking back for the history of the svn repo:

Hi Bill,

David Moore has invited you to a Dropbox shared folder called "SIPNET_repositoryFiles", and left you this message:

"Folks,
Here is a folder where you can post any SIPNET files for specific sites and/or other useful files for creating SIPNET files or processing SIPNET data.
Over the next few months we will move these files to the SIPNET repository (managed by John)."

@billsacks I have a different version of SIPNET, unfortunately it is just a dump of source files no svn folders. I believe this is a snapshot from 2016-06-02. You can find it at http://isda.ncsa.illinois.edu/~kooper/PEcAn/sipnet/sipnet_r136.tar.gz (looking at the r136 I think this might be revision 136 so maybe 136 commits, not really sure anymore about svn revisions)

@robkooper Thanks for this. I took a very quick look and it looks like your version has a lot of directory cleanup, as well as a bit in the way of new functionality.

The svn repo I used to create the GitHub repo only went up to r81. I can dig a bit more to see if I happen to have a more recent version of the full repo, but I have very low hopes for that. Assuming I don't have anything more recent, that leaves us with two options that I can see:

(1) Get a more recent version from @jmzobitz . John, it sounds like you might be back on the hook here after all for trying to dig up the latest version of the repo. Please let us know if this still seems to be at all a possibility.

(2) I could make a single commit that applies all of the changes between r81 and r136. If I do this, I'd like to give attribution if possible, so would be happy to hear from anyone who has a sense of who gets the credit for these changes.

Thoughts or any other suggestions?

@billsacks @robkooper @jmzobitz I think I have a more recent pull from John's repo:

Path: .
Working Copy Root Path: /fs/data5/pecan.models/SIPNET/trunk
URL: http://turing.augsburg.edu/svn/sipnet/trunk
Relative URL: ^/trunk
Repository Root: http://turing.augsburg.edu/svn/sipnet
Repository UUID: c88c3319-8f09-0410-a662-fffcf0821eeb
Revision: 135
Node Kind: directory
Schedule: normal
Last Changed Author: zobitz
Last Changed Rev: 135
Last Changed Date: 2012-03-01 22:35:10 -0500 (Thu, 01 Mar 2012)

I'm not great with svn, and I don't have time to figure it out, but if someone told me what to do I could send them this code (I'm assuming it involves more than just emailing you a tarball of the directory). Alternatively, if @robkooper wanted to take a crack the Working Copy Root Path above is on our BU pecan machines, which he has access to.

@mdietze unfortunately, unlike with git, I'm pretty sure that the centralized nature of svn means that you can't get the history from a checkout: you need the server for that. (You can try doing svn log from that directory: I strongly suspect it will fail based on the fact that turing.augsburg.edu appears to no longer be running.)

@jmzobitz I have looked in a few more places, and can't find any more recent backups of the repository than r81. So I think maintaining more recent history will depend on whether you're able to find anything there.

Hi all - I checked with colleagues here and found out today that turing was not backed up before it crashed. * sigh *.

The good news is that I think the SVN of SIPNET was stable-ish, meaning the latest numbered revision that people have is probably the best one. If I recall correctly the version used in fluxcourse was fairly constant through the years.

@jmzobitz or others: Given other obligations I have coming up (meetings, travel and a bunch of stuff I need to get done before that), I'd like to ask that someone else take the final step of bringing in the changes between r81 and r136. What I imagine is that someone will make one or multiple commits that apply the changes from r81 to r136, such that the history up to r81 is preserved (which should happen automatically if you do this by adding commits on top of the repo I set up) and the final state looks exactly like the r136 tarball. (With bonus points awarded for proper authorship attribution and commit message(s) for the changes between r81 and r136.)

These final changes can then be submitted as a PR into my repo, or I can give someone else write permission to the temporary repo I set up.

Volunteers?

Created a pull request https://github.com/billsacks/sipnet/pull/1 that adds the changes to the code from r81 to r136. The majority seems to be addition of data.

Thanks a lot, @robkooper ! I've started some discussion in that PR, and I'd suggest that we move any discussion of the r81 -> r136 update to there, then bounce back here when we're ready to move sipnet to PECAN.

Thanks to @robkooper , the master branch on https://github.com/billsacks/sipnet now matches the latest version of sipnet we were able to find (r136). History is preserved up through r81, then @robkooper introduced a few extra bulk commits to pull in the changes from r82 through r136. So I think this is ready to be moved to its final home in PecanProject. Would someone like to temporarily give me permissions to do this move? Alternatively, I can make someone else an admin for https://github.com/billsacks/sipnet so that you can do the move yourself.

Added you as admin, let me know when you finished transferring.

Okay, thanks, I'm done transferring it.

• Added two tags r81 and r136 as releases.
• Created team sipnet that has write permissions to repository
• Added @mdietze and @billsacks to sipnet team (and made them maintainers of team so they can add additional people)

@istfer you can create a pull request against sipnet (fork sipnet, branch, edit, push, pull request)

Hooray! Thanks everyone for making this happen! I'm sure it will increase the number of studies that use SIPNET too!

This is excellent, I agree!

@robkooper thanks again for this. I have no need to be a maintainer of the sipnet repository long-term, since I don't imagine being active in its future development.

right now it is @mdietze, @billsacks and @robkooper. If there are others that might be a maintainer they can be added to the sipnet team. Otherwise we can leave it as is and Mike and I will do most of the pull requests (until we find another volunteer).

Would @jmzobitz also be interested in being a maintainer?

@istfer Thanks for asking. In the short term, no. I would like to develop an R version of SIPNET, but I won't have time to work on this in the short term (6 months out)

Closing issue since code is now available https://github.com/pecanproject/sipnet