Pecan: Enchancement for remote.copy.from
Description
Is your feature request related to a problem? Please describe.
With more and more people using remote clusters for performing compute-intensive PEcAn runs/analyses, there is now a need for not syncing everything back from the remote. This would also help with space issues on local clusters (like we experience at BU).
Proposed Solution
Describe the solution you'd like
I was thinking of extending the existing remote.copy.from function such that it accepts one or two additional arguments. For example, an include and a rule argument (for file filtering rules), and when they are not NULL the rsync command is modified accordingly within the function.
Alternatives Considered
Describe alternatives you've considered
Alternatively a whole another function (or functions) can be added to the remote package, like remote.copy.from.with.include / remote.copy.from.with.rule
istfer
All 6 comments
e.g. @femeunier uses exclude https://github.com/femeunier/pecan/blob/d12b902cbc244983a6fdb2eee0d13ab9216149eb/base/remote/R/remote.copy.from.R#L22
I specify bunch of includes and exclude everything else, e.g. "--include=pecan.pda*", "--include=history*", "--exclude=*"
Tagging people who might be interested @femeunier @para2x @ashiklom @mdietze @robkooper
istfer
on 17 May 2019
What if you make this as an argument which people can start with whatever flags they need ? I mean not hard coded.
para2x
on 17 May 2019
Oh yeah, it definitely shouldn't be hardcoded I was just giving examples
istfer
on 17 May 2019
Oh yeah, it definitely shouldn't be hardcoded I was just giving examples
Very helpful.
para2x
on 17 May 2019
Would something like this work?
remote.copy.from <- function(host, src, dst, options=c(), delete = FALSE, stderr = FALSE) {
args <- c("-az", "-q", options)
robkooper
on 17 May 2019
great, yes I think that would be the most flexible version
istfer
on 17 May 2019
Related issues
serbinsh
·
12Comments
ashiklom
·
9Comments
dlebauer
·
5Comments
tonygardella
·
5Comments
istfer
·
8Comments
Most helpful comment
Would something like this work?