Restic: Exclude patterns not being applied
Output of restic version
restic 0.6.1 (v0.6.1-126-gb0fb95df)
compiled with go1.8.3 on darwin/amd64
How did you start restic exactly? (Include the complete command line)
restic backup --exclude-file ~/exclude.txt ~
What backend/server/service did you use?
sftp to LAN
Expected behavior
Skip the files matching the pattern in exclude.txt
Actual behavior
Files were included unless I drastically loosened the pattern to the point where the pattern is too ambiguous.
Steps to reproduce the behavior
Using an exclude file like this:
.[^.]*
/matt/Applications
/matt/Dropbox
/matt/Downloads
/matt/VirtualBox VMs
*.vmdk
My goal is to skip dotfiles and a few specific folders in my $HOME directory (named matt) (but not every folder named "Applications" for example). Oh, and any .vmdk files (they can be huge!)
When I relaxed the patterns to this:
.[^.]*
Applications
Dropbox
Downloads
VirtualBox VMs
*.vmdk
it worked (meaning it skipped the folders) but this is too ambiguous, as I don't want to skip all folders named Downloads, for example.
Do you have any idea what may have caused this?
Probably me still not understanding exclude patterns correctly. :smile: I did read #1005 and I guess I'm confused if absolute paths are required for --exclude or if repo-absolute paths are required like with --include.
mholt
All 4 comments
Update: Okay, replacing /matt/ with $HOME/ seemed to work, although I find the inconsistency between --exclude and --include absolute paths a little odd/surprising.
mholt
on 20 Jun 2017
Usually $HOME/blabla is /home/matt/blabla or did you change something on your system?
ibib
on 20 Jun 2017
Sigh, yes, you are absolutely right. The exclude patterns are absolute in regards to what you backup:
The pattern /home/matt/*.go matches all Go files in your home when you run restic backup /home/matt. Due to the way the archiver works (see #549, bad design decision we need to correct), the /home/ prefix is dropped and the top-level path component in the snapshot is /matt. So for restore, you'd need to use an include pattern of /matt/*.go to match the same files. I know it's bad, I'll correct it eventually :)
So, this is the corrected exclude file list for your use case:
.[^.]*
/home/matt/Applications
/home/matt/Dropbox
/home/matt/Downloads
/home/matt/VirtualBox VMs
*.vmdk
fd0
on 20 Jun 2017
@ibib On my Mac, my home directory is /Users/matt, but since --include uses a repository-absolute path (like restic ls shows), I figured the proper absolute path would be /matt/....
@fd0 Ah! That makes more sense, thanks for explaining. So --exclude when used with restic backup is absolute to the file system, and --include (used with restic restore) is absolute to the repository. In other words, the patterns are absolute to where the files are coming from.
Thanks so much for the explanation. Now I understand better why 549 is an issue you want to fix. :smile:
mholt
on 20 Jun 2017
Related issues
christian-vent
·
3Comments
RafaelAybar
·
3Comments
stevesbrain
·
3Comments
ikarlo
·
4Comments
fd0
·
4Comments
Most helpful comment
Sigh, yes, you are absolutely right. The exclude patterns are absolute in regards to what you backup:
The pattern
/home/matt/*.gomatches all Go files in your home when you runrestic backup /home/matt. Due to the way the archiver works (see #549, bad design decision we need to correct), the/home/prefix is dropped and the top-level path component in the snapshot is/matt. So for restore, you'd need to use an include pattern of/matt/*.goto match the same files. I know it's bad, I'll correct it eventually :)So, this is the corrected exclude file list for your use case: