Restic: Allow mixing include and exclude patterns for backup and restore
This is a followup issue for #226. Up to now it is only possible to specify exclude patterns for backup and restore.
There are uses cases that aren't covered, for example: A user would like to backup her home directory at /home/user, excluding anything from the directory work except C source files.
Are there any other uses I haven't thought of?
Implementation/User Interface: Allow specifying both --include and --exclude patterns on the command line, which fill a common list (order does matter here). For each dir/file, check all patterns in the list. The action (exclude or include) of the last mattching pattern wins. The default action (which will also be used if neither include nor exclude patterns are specified) is "include".
This already contains a corner case: Should restic walk /home/user/work at all?
I think it should not. Instead, if this is a desirable behaviour, require the user to add a more specific pattern to signal that the excluded directory should also be walked, e.g. restic backup --exclude /home/user/work --include /home/user/work/**/*c. /home/user.
A better user interface would be to allow specifying a file to read the patterns from (--pattern-file or something like that). In this file, all lines starting with # are comments, empty lines ignored, all other lines must start with either + (include) or - (exclude) following a space character and a pattern. For the use case above a filter file would look like this:
# filter out everything from work, but include c source code files
- /home/user/work
+ /home/user/work/**/*.c
This issue can be closed once a definitive solution for having both include and exclude patterns is implemented.
fd0
All 17 comments
Another possibility would be to read an exact list of files to backup from a file (or from stdin). Then people could just use whatever tools they want (find, grep, etc.) to build their list and "pipe" that list to restic.
fw42
on 18 Jul 2015
I should read the whole text before posting a reply. Sorry.
fw42
on 18 Jul 2015
This is a different issue, taking the list of files/dirs to backup from stdin instead of command line arguments. Do you think it's valuable to have that? If so, could you add an issue?
fd0
on 18 Jul 2015
Just a reminder for myself: This issue is about include filters for backup, restore already has them.
fd0
on 10 Nov 2015
You should maybe just copy rsync's --exclude/--include/--exclude-from syntax, they have 20 years of experience :-)
(at least please add some examples of the current syntax to the user guide as it's unclear if "/foo" and "foo" is thesame or if "*.c" is supported.
lathspell
on 24 Jan 2016
Ah, thanks for the comment, I've added an issue for the missing examples in the manual: https://github.com/restic/restic/issues/396
To be honest, I don't like the filter syntax from rsync at all, because the rules are way too complex. But we'll see what we can come up with.
fd0
on 25 Jan 2016
Any updates? It is definitely a must-have option. Currently it is not even possible to read the file list to back up from the stdin:
$ find -name '*.go' | restic backup --files-from -
open -: no such file or directory
whereas it could be written as
restic backup --exclude '*' --include '*.go'
opennota
on 3 Feb 2017
Uhm, reading the list of files from stdin can be achieved by calling restic as follows:
$ find -name '*.go' | restic backup --files-from /dev/stdin
If you like, I'd accept a PR that adds handling - for --files-from. :)
fd0
on 3 Feb 2017
@opennota would you mind describing your use case? It'd be interesting for us.
fd0
on 3 Feb 2017
The dash (-) issue is tracked as #769.
fd0
on 3 Feb 2017
@fd0
Not much of a use case. I just want to back up only the files with certain extensions without using a temporary file for the list.
If you like, I'd accept a PR that adds handling - for --files-from. :)
I'll see what I can do.
opennota
on 3 Feb 2017
You can kind of emulate this behavior using sed and named pipes:
restic --exclude-file <(sed -n 's/^- \(.*\)/\1/p' files.list) --files-from <(sed -n 's/^+ \(.*\)/\1/p' files.list)
Lines starting with - are excluded and those with + are included.
runfalk
on 19 Mar 2018
I think a good model is what borg recently implemented for --pattern and --patterns-from
https://borgbackup.readthedocs.io/en/stable/usage/help.html#borg-help-patterns
Not so much the different style selectors, but the options to specify root paths, include rules, exclude rules, and no-recurse exclude rules in a file.
dgasaway
on 17 Jul 2018
Why not just use the standard ignore format that .gitignore uses? eg.
# ignore everything
*
# include $HOME/.local
!$HOME/.local
blacksails
on 29 Jan 2019
The --include is scheduled to be implemented?
Sound like --include and --exclude cannot be implemented together, or at least would be a hierarchy of precedence with one on other...
tnatanael
on 26 Apr 2019
Providing a list of files with --files-from doesn't totally solve the problem as the snapshots subcommand will display gigantic list of files and forget subcommand doesn't work as expected.
My use case is backing up my home and I have a list of paths with some exclusion and some exceptions for the exclusion. The base list of paths contains already a hundred of items. As everything is under $HOME, I would expect to be able to say something like --exclude=** --include=~/path1 --include=~/path2 --exclude=~/path2/something --exclude=*~. So, to determine if a path should be included it should be matched against each --exclude and --include in the right order and the last matching wins.
vincentbernat
on 14 Jun 2019
I think @vincentbernat's PR is an effective solution to this requirement.
_Summary: Allow gitignore-style negative patterns to specify exclude rules for both backup and restore._
I've been using it effectively for several weeks now.
Notably, this enables me to simplify my snapshots listing, which was quite verbose previously:
b951f6a2 2019-06-15 11:30:18 elvandar manual /Users/daniel/Desktop
/Users/daniel/Documents
<lots more...>
to:
d0c0bed1 2019-06-18 08:20:57 elvandar /Users/daniel
Also, I've implemented an effective (not perfect but working) solution providing continuous backup using this feature (will share details shortly).
danielsharvey
on 3 Jul 2019
Related issues
McKael
·
4Comments
christian-vent
·
3Comments
RafaelAybar
·
3Comments
shibumi
·
3Comments
e2b
·
4Comments
Most helpful comment
Why not just use the standard ignore format that
.gitignoreuses? eg.