Restic: Very slow backup of SSHFS filesystems -- need ability to skip inode-based comparison
Output of restic version
restic 0.8.2
How did you run restic exactly?
% restic -r /tmp/restictest backup $SSHFSMOUNT/directory
The backup directory is a sshfs mounted filesystem.
The command is run several times to do several snapshots.
In mly case there is a huge amount of data (1TB+) in the remote directory.
What backend/server/service did you use to store the repository?
Local ext4 filesystem.
Expected behavior
I expect the first backup to be slow and transfer much data over LAN, while the next backups should be pretty fast and should not use much bandwidth.
Actual behavior
All backups take hours (if not days) even if no files have changed.
Steps to reproduce the behavior
- run a backup
- unmount sshfs filesystem
- re-mount sshfs filesystem
- run a new backup
It is not 100% reproducible but even with a small amount of data I could reproduce it.
The SFTP serrver logs show that files are completely retrieved even when they did not change.
Do you have any idea what may have caused this?
Yes: restic compares inodes to double-check if files have been modified (debug message "timestamp, size or inode changed", restic/node.go:551restic.(*Node).IsNewer11node).
However, inodes might change across filesystem mounts with sshfs (and probaly some other filesystems).
Do you have an idea how to solve the issue?
Commenting out the inode check solved the problem for me.
I'd like to have a way to disable this check; maybe a command-line flag?
Did restic help you or made you happy in any way?
Sure, it's a nice piece of software! I'm all the more pleased since I have found a workaround...
Keep up the good work!
McKael
All 4 comments
Thanks for the report, this is indeed caused by restic detecting that files have changed based on the inode. For fuse-based file systems, this check is not great, we should instead only check the timestamps and the file size.
In principle this could also be detected automatically (by looking at the file system name, and keeping a black list of know-inode-unstable filesystems) so we may not even need a command-line flag.
fd0
on 19 Feb 2018
In principle this could also be detected automatically (by looking at the file system name, and keeping a black list of know-inode-unstable filesystems) so we may not even need a command-line flag.
How do you envision this? I could give it a shot...
ebastos
on 26 Apr 2018
2205 was merged and is in 0.9.5, this should be closed. :wink:
cyphar
on 28 Apr 2019
You're right, thanks for the hint!
fd0
on 28 Apr 2019
Related issues
shibumi
·
3Comments
mholt
·
4Comments
fd0
·
3Comments
christian-vent
·
3Comments
rakor
·
5Comments
Most helpful comment
Thanks for the report, this is indeed caused by restic detecting that files have changed based on the inode. For fuse-based file systems, this check is not great, we should instead only check the timestamps and the file size.
In principle this could also be detected automatically (by looking at the file system name, and keeping a black list of know-inode-unstable filesystems) so we may not even need a command-line flag.