Restic: handle large prune much more efficent

With performance for restores for large repos/repos with large files now resolved by @ifedorenko's out-of-order branch, it looks like this is the next boulder for using restic in multi-terabyte environments where the repo isn't on local disk and isn't being accessed via the REST server on a loopback interface.

Currently, an empty prune (pruning a snapshot that was 100% identical to a previous snapshot) against a repo in an AZ-local S3 bucket on high-end AWS instances with 10gbit/sec theoretical bandwidth runs at:

1) building new index -- ~160 packs/second
2) finding data still in use -- 56 seconds total
3) rewriting packs -- ~3 packs/second

160 packs/second is slow, but still tolerable (~80 minute run-time against a 3TB repo).

But the rewrite @ 3 packs/second will run for almost 10 hours, even for my noop prune (will delete 0 packs and rewrite 111098 packs, this frees 180.699 GiB). For a meaningful prune on a large repo, you freeze out new backups for 24+ hours.

It looks like the pack rewrites currently happen in a single-thread, so allowing that to run across multiple workers might help quite a bit, even if the current copy then purge approach is maintained.

Personally, I would not spend time optimizing current blocking prune implementation, I think non-blocking prune is better long-term solution.

I've been working on this recently. Here's what I've done:

• Load the existing index instead of scanning all packs (and then scan any packs that weren't in the index)
• Parallelized scanning the snapshots for used blobs
• Parallelized rewriting partially used packs
• Write the new index using the index info already in memory

I'm currently working to figure out the level of parallelism to use for rewriting partially used packs (I'm planning to base this on the configured number of connections for the backend). I also need to do a lot more testing in various error scenarios.

Here are some performance numbers (using a repository with 875 GiB of data, about 180,000 packs, and 36 snapshots, using a loopback rest server as the backend):

• Current code:
  
  
  
  • 35-40 minutes (each) to build the index at the beginning and end of the prune (70-80 minutes total)
  
  
  • 4-5 minutes to find used blobs
  
  
  • A few seconds to rewrite partially used packs
  
  
• My changes so far:
  
  
  
  • A few seconds to load the existing index (somewhat longer if it needs to scan unindexed packs)
  
  
  • Under 2 minutes to find used blobs
  
  
  • A few seconds to write the new index
  
  

I'm also planning to set up a generated test case that will involve a lot more pack rewriting.

Courtney:

Sounds super promising! Wanted to confirm that this is the branch you are working in? I'm happy to test.

https://github.com/cbane/restic/tree/prune-aggressive

No, that branch is part of the fork from the main repository. I haven't pushed my changes anywhere public yet. I should be able to push my work-in-progress version to GitHub in a few days so you can try it out.

OK, I have a version that should be ready for other people to try out. It's on this branch: https://github.com/cbane/restic/tree/prune-speedup

Current limitations:

• I haven't implemented any automatic setting of the number of repack workers. For now, set the environment variable RESTIC_REPACK_WORKERS to the number of workers you want to use. It will default to 4 if it's not set.
• I need to work on the error handling when repacking. I didn't make any real changes from the existing single-threaded repacking; I need to look over the various error cases and make sure that doing the repacking in parallel doesn't cause any problems.

Um, that looks amazing. Thank you for your work!

I have tested this a bit with a copy of 3TB+ repo in Amazon S3 and so far it looks amazing. A repack prune that would have taken weeks now completes in under an hour, and that's using relatively slow EBS as tmp space.

A real game changer here! Great work, @cbane!

Eek, I realized I mistimed the run.

One area that is still single threaded and looks like it could benefit from parallelization is the "checking for packs not in index" step -- that can still take 3-4 hours in multi-terabyte repos -- but this is still a massive, massive improvement, thank you!

@cbane I wasn't able to open an issue against your fork, so let me know if there's a better place for these.

During another test run, the repack failed at the very end (rewriting the last pack), running with 32 workers:

found 1396709 of 2257203 data blobs still in use, removing 860494 blobs
will remove 0 invalid files
will delete 119301 packs and rewrite 88485 packs, this frees 896.269 GiB
using 32 repack workers
Save(<data/c136027b25>) returned error, retrying after 723.31998ms: client.PutObject: Put https://ak-logical-db-backup.s3.dualstack.us-west-1.amazonaws.com/xxxxx: Connection closed by foreign host https://ak-logical-db-backup.s3.dualstack.us-west-1.amazonaws.com/xxxx. Retry again.
Save(<data/09d4692900>) returned error, retrying after 538.771816ms: client.PutObject: Put https://ak-logical-db-backup.s3.dualstack.us-west-1.amazonaws.com/xxxxx: Connection closed by foreign host https://ak-logical-db-backup.s3.dualstack.us-west-1.amazonaws.com/xxxxx. Retry again.
Save(<data/23d0d4f842>) returned error, retrying after 617.601934ms: client.PutObject: Put https://ak-logical-db-backup.s3.dualstack.us-west-1.amazonaws.com/xxxx: Connection closed by foreign host https://ak-logical-db-backup.s3.dualstack.us-west-1.amazonaws.com/xxxx. Retry again.
[10:02] 100.00%  88484 / 88485 packs rewritten
panic: reporting in a non-running Progress

goroutine 386596 [running]:
github.com/restic/restic/internal/restic.(*Progress).Report(0xc42011c2c0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0)
        internal/restic/progress.go:122 +0x242
github.com/restic/restic/internal/repository.Repack.func2(0x8, 0xe17f58)
        internal/repository/repack.go:66 +0x136
github.com/restic/restic/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc4389246c0, 0xc56f509160)
        vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x57
created by github.com/restic/restic/vendor/golang.org/x/sync/errgroup.(*Group).Go
        vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66

I have a new version available, at the same branch as before. I also rebased against master.

Here are the main changes from the previous version:

• Converted the repacking from having each worker do all stages of repacking a single pack to a pipeline.
• Fixed the reported crash at the end of repacking.
• Repacking now auto-scales the number of workers for the pipeline stages based on the number of CPUs and the configured connection limit for the backend. (The RESTIC_REPACK_WORKERS environment variable is no longer used.)
• Minor tweaks to finding used blobs.
• Parallelized the scanning of unknown packs.

I still want to do some more work on finding used blobs. Currently, each snapshot is processed by a single worker. This can leave CPU resources unused if there are fewer snapshots than CPUs, or if there are major size differences between snapshots. I would like to have it spread sub-tree processing across different workers; I think I know how to do this, I just need to actually implement it.

I'd lean toward continuing to discuss things on this issue (or the future pull request for this), so that everything stays here in the main repository instead of spread out.

Just tested. It resolves the crash at the end of repacking and works really, really well.

The only additional place that could benefit from increased parallelism is the deletion of packs, which is currently single threaded.

This bites particularly hard during the first prune of a repo that was not previously prune-able, since there are a _lot_ of packs that need deletion.

Even with single threaded deletion, however, a daily forget/prune against a 1.7TB, 356k pack repository that rewrites 14.7k packs and deletes 33k packs now takes just under 20 minutes.
Previously, it was impossible to prune at all.

Excellent work, thank you!

OK, I have another version available. The only real change this time is it now deletes unused packs in parallel (plus a few minor tweaks to some previous changes). I implemented the modified snapshot scanning, but it didn't give enough of a speedup and there wasn't a good way to report progress to the user, so I removed it again.

I'm planning to open a pull request for this soon, assuming that nothing broke. (I'll clean up the history first, though.) @fd0, do you want to take a look at this first?

Worked great in our test run. Rewrote 30k packs in 225 seconds, deleted 73k packs in 50 seconds.

Total runtime against a 1.74TiB repo in S3 with 32 surviving snapshots was just over 6 minutes.

Brilliant work.

@cbane I tried your branch https://github.com/cbane/restic/tree/prune-speedup

but this is the error I recive :(

root@srv ~/restic-backups # restic --no-cache prune
repository 49b87c6a opened successfully, password is correct
listing files in repo
loading index for repo
[0:28] 1.01%  30 / 2982 index files
pack cbbebd8d already present in the index
github.com/restic/restic/internal/index.(*Index).AddPack
    internal/index/index.go:266
github.com/restic/restic/internal/index.Load.func1
    internal/index/index.go:230
github.com/restic/restic/internal/repository.(*Repository).List.func1
    internal/repository/repository.go:640
github.com/restic/restic/internal/backend.(*RetryBackend).List.func1.1
    internal/backend/backend_retry.go:133
github.com/restic/restic/internal/backend/rest.(*Backend).listv2
    internal/backend/rest/rest.go:423
github.com/restic/restic/internal/backend/rest.(*Backend).List
    internal/backend/rest/rest.go:358
github.com/restic/restic/internal/backend.(*RetryBackend).List.func1
    internal/backend/backend_retry.go:127
github.com/cenkalti/backoff.RetryNotify
    vendor/github.com/cenkalti/backoff/retry.go:37
github.com/restic/restic/internal/backend.(*RetryBackend).retry
    internal/backend/backend_retry.go:36
github.com/restic/restic/internal/backend.(*RetryBackend).List
    internal/backend/backend_retry.go:126
github.com/restic/restic/internal/repository.(*Repository).List
    internal/repository/repository.go:634
github.com/restic/restic/internal/index.Load
    internal/index/index.go:202
main.pruneRepository
    cmd/restic/cmd_prune.go:202
main.runPrune
    cmd/restic/cmd_prune.go:128
main.glob..func18
    cmd/restic/cmd_prune.go:28
github.com/spf13/cobra.(*Command).execute
    vendor/github.com/spf13/cobra/command.go:762
github.com/spf13/cobra.(*Command).ExecuteC
    vendor/github.com/spf13/cobra/command.go:852
github.com/spf13/cobra.(*Command).Execute
    vendor/github.com/spf13/cobra/command.go:800
main.main
    cmd/restic/main.go:86
runtime.main
    /snap/go/3947/src/runtime/proc.go:200
runtime.goexit
    /snap/go/3947/src/runtime/asm_amd64.s:1337

@antetna It looks like your repository has multiple index files that cover the same packs. I don't know how that can happen, but I've added a test case for it to the test suite, and I can reproduce your error. I'll work on fixing it.

@antetna OK, I just pushed a new version to the same branch (not fast-forward) that works with my duplicate index test case. Could you give it a try?

I'm planning on opening a pull request with the current version of this branch soon, unless anybody else notices any problems.

Thank you very much @cbane! The standard prune took around ~55 hours to prune my ~860GB 12000+ snapshot repo. With your more aggressive parallel approach it dropped it down to just over 3 hours.

Howdy @cbane, amazing work!

Running this PR here on Debian 9 amd64, compiled with Go 1.12.1, getting the following error after 220 minutes running on my 30TB repo:

checking for packs not in index
[0:52] 16.45%  178 / 1082 packs
[5:23] 100.00%  1082 / 1082 packs
repository contains 3213929 packs (259446787 blobs) with 15.025 TiB
processed 259446787 blobs: 30090 duplicate blobs, 4.906 GiB duplicate
load all snapshots
find data that is still in use for 3 snapshots
[0:04] 0.00%  0 / 3 snapshots
tree 6f144a19eaae0e81518b396bfdfc9dd5c6c035bdba28c6a90d6f70e692aa1c55 not found in repository
github.com/restic/restic/internal/repository.(*Repository).LoadTree
        internal/repository/repository.go:707
github.com/restic/restic/internal/restic.FindUsedBlobs.func3
        internal/restic/find.go:52
github.com/restic/restic/internal/restic.FindUsedBlobs.func3
        internal/restic/find.go:74
github.com/restic/restic/internal/restic.FindUsedBlobs.func3
        internal/restic/find.go:74
github.com/restic/restic/internal/restic.FindUsedBlobs.func3
        internal/restic/find.go:74
github.com/restic/restic/internal/restic.FindUsedBlobs.func4
        internal/restic/find.go:89
gopkg.in/tomb%2ev2.(*Tomb).run
        vendor/gopkg.in/tomb.v2/tomb.go:163
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1337

How should I recover from this?

Thanks,
-- Durval.

@DurvalMenezes It looks like your repository is missing some pack files. Have you pruned it before? Does restic check succeed? If it doesn't, it should give you a more detailed idea of what's wrong.

You might be able to recover somewhat by running restic rebuild-index and then running a new backup. If anything in the missing packs is still available in the location being backed up, a new backup will add it to the repository again. This might make your existing backups work again. If that doesn't fix it, I think that you will need to forget the affected snapshots before you can do a prune.

Thanks for the reply, @cbane. More, below:

Have you pruned it before?

Nope, this is my first prune. Long story short: my repo has ~95 snapshots from 3 local dirtrees on my NAS; those 3 dirtrees total ~30TB and ~60M files/subdirs, and has been taking longer and longer to backup, to the point that backing up just 24 hours of new data (under 10GB) was taking over 24 hours to run. The advice on the forum was to run restic forget (which I did, leaving only the last 3 snapshots) and then restic prune, which I did first using restic 0.9.5 (but it aborted after ~24 hours due to OOM). I then upgraded the machine(a VM on Google Compute Cloud) to twice the RAM, and then used your PR, which gave the above error.

Does restic check succeed? If it doesn't, it should give you a more detailed idea of what's wrong.

I'm running restic check for the last 90 hours (also using your PR), and it has so far given me this output: restic-check-PARTIAL_output.txt

As noted at the end of the above file, restic check has displayed its latest message ("check snapshots, trees and blobs") over 3 days ago... I'm starting to wonder it's stuck and will never finish :-/ In fact, an "strace" on the process shows it opening the same local cache file over and over again:

```date; strace -t -f -p 2608 -e openat 2>&1 | grep openat | egrep -v unfinished\|resumed | head
Tue Jul 23 00:41:59 UTC 2019
[pid 26508] 00:41:59 openat(AT_FDCWD, "/tmp/restic-check-cache-370688148/abb62ab49b950e4b2ab5eb82c4386a9b199a08eb6fd93bdaccd0e4dbe10d29a2/data/53/53d242f8d6444bc2bd49e6689b4c7688fec8996c5e86863bf146a9eb28f54b5d", O_RDONLY|O_CLOEXEC) = 5
[pid 2608] 00:41:59 openat(AT_FDCWD, "/tmp/restic-check-cache-370688148/abb62ab49b950e4b2ab5eb82c4386a9b199a08eb6fd93bdaccd0e4dbe10d29a2/data/53/53d242f8d6444bc2bd49e6689b4c7688fec8996c5e86863bf146a9eb28f54b5d", O_RDONLY|O_CLOEXEC) = 10
[pid 2615] 00:41:59 openat(AT_FDCWD, "/tmp/restic-check-cache-370688148/abb62ab49b950e4b2ab5eb82c4386a9b199a08eb6fd93bdaccd0e4dbe10d29a2/data/53/53d242f8d6444bc2bd49e6689b4c7688fec8996c5e86863bf146a9eb28f54b5d", O_RDONLY|O_CLOEXEC) = 5
[pid 5482] 00:41:59 openat(AT_FDCWD, "/tmp/restic-check-cache-370688148/abb62ab49b950e4b2ab5eb82c4386a9b199a08eb6fd93bdaccd0e4dbe10d29a2/data/53/53d242f8d6444bc2bd49e6689b4c7688fec8996c5e86863bf146a9eb28f54b5d", O_RDONLY|O_CLOEXEC) = 5
[pid 2615] 00:41:59 openat(AT_FDCWD, "/tmp/restic-check-cache-370688148/abb62ab49b950e4b2ab5eb82c4386a9b199a08eb6fd93bdaccd0e4dbe10d29a2/data/53/53d242f8d6444bc2bd49e6689b4c7688fec8996c5e86863bf146a9eb28f54b5d", O_RDONLY|O_CLOEXEC) = 5
[pid 3688] 00:41:59 openat(AT_FDCWD, "/tmp/restic-check-cache-370688148/abb62ab49b950e4b2ab5eb82c4386a9b199a08eb6fd93bdaccd0e4dbe10d29a2/data/53/53d242f8d6444bc2bd49e6689b4c7688fec8996c5e86863bf146a9eb28f54b5d", O_RDONLY|O_CLOEXEC) = 5
[pid 5482] 00:41:59 openat(AT_FDCWD, "/tmp/restic-check-cache-370688148/abb62ab49b950e4b2ab5eb82c4386a9b199a08eb6fd93bdaccd0e4dbe10d29a2/data/53/53d242f8d6444bc2bd49e6689b4c7688fec8996c5e86863bf146a9eb28f54b5d", O_RDONLY|O_CLOEXEC) = 10
[pid 2608] 00:41:59 openat(AT_FDCWD, "/tmp/restic-check-cache-370688148/abb62ab49b950e4b2ab5eb82c4386a9b199a08eb6fd93bdaccd0e4dbe10d29a2/data/53/53d242f8d6444bc2bd49e6689b4c7688fec8996c5e86863bf146a9eb28f54b5d", O_RDONLY|O_CLOEXEC) = 11
[pid 2638] 00:41:59 openat(AT_FDCWD, "/tmp/restic-check-cache-370688148/abb62ab49b950e4b2ab5eb82c4386a9b199a08eb6fd93bdaccd0e4dbe10d29a2/data/53/53d242f8d6444bc2bd49e6689b4c7688fec8996c5e86863bf146a9eb28f54b5d", O_RDONLY|O_CLOEXEC) = 5
[pid 2638] 00:41:59 openat(AT_FDCWD, "/tmp/restic-check-cache-370688148/abb62ab49b950e4b2ab5eb82c4386a9b199a08eb6fd93bdaccd0e4dbe10d29a2/data/53/53d242f8d6444bc2bd49e6689b4c7688fec8996c5e86863bf146a9eb28f54b5d", O_RDONLY|O_CLOEXEC) = 5

And then, almost 10 hours later: 

Tue Jul 23 10:14:27 UTC 2019
[pid 2632] 10:14:27 openat(AT_FDCWD, "/tmp/restic-check-cache-370688148/abb62ab49b950e4b2ab5eb82c4386a9b199a08eb6fd93bdaccd0e4dbe10d29a2/data/53/53d242f8d6444bc2bd49e6689b4c7688fec8996c5e86863bf146a9eb28f54b5d", O_RDONLY|O_CLOEXEC) = 5
[pid 2639] 10:14:28 openat(AT_FDCWD, "/tmp/restic-check-cache-370688148/abb62ab49b950e4b2ab5eb82c4386a9b199a08eb6fd93bdaccd0e4dbe10d29a2/data/53/53d242f8d6444bc2bd49e6689b4c7688fec8996c5e86863bf146a9eb28f54b5d", O_RDONLY|O_CLOEXEC) = 5
[pid 2634] 10:14:28 openat(AT_FDCWD, "/tmp/restic-check-cache-370688148/abb62ab49b950e4b2ab5eb82c4386a9b199a08eb6fd93bdaccd0e4dbe10d29a2/data/53/53d242f8d6444bc2bd49e6689b4c7688fec8996c5e86863bf146a9eb28f54b5d", O_RDONLY|O_CLOEXEC) = 5
[pid 2613] 10:14:28 openat(AT_FDCWD, "/tmp/restic-check-cache-370688148/abb62ab49b950e4b2ab5eb82c4386a9b199a08eb6fd93bdaccd0e4dbe10d29a2/data/53/53d242f8d6444bc2bd49e6689b4c7688fec8996c5e86863bf146a9eb28f54b5d", O_RDONLY|O_CLOEXEC) = 5
[pid 2634] 10:14:28 openat(AT_FDCWD, "/tmp/restic-check-cache-370688148/abb62ab49b950e4b2ab5eb82c4386a9b199a08eb6fd93bdaccd0e4dbe10d29a2/data/53/53d242f8d6444bc2bd49e6689b4c7688fec8996c5e86863bf146a9eb28f54b5d", O_RDONLY|O_CLOEXEC) = 5
[pid 3688] 10:14:28 openat(AT_FDCWD, "/tmp/restic-check-cache-370688148/abb62ab49b950e4b2ab5eb82c4386a9b199a08eb6fd93bdaccd0e4dbe10d29a2/data/53/53d242f8d6444bc2bd49e6689b4c7688fec8996c5e86863bf146a9eb28f54b5d", O_RDONLY|O_CLOEXEC) = 5
[pid 2617] 10:14:28 openat(AT_FDCWD, "/tmp/restic-check-cache-370688148/abb62ab49b950e4b2ab5eb82c4386a9b199a08eb6fd93bdaccd0e4dbe10d29a2/data/53/53d242f8d6444bc2bd49e6689b4c7688fec8996c5e86863bf146a9eb28f54b5d", O_RDONLY|O_CLOEXEC) = 5
[pid 3688] 10:14:28 openat(AT_FDCWD, "/tmp/restic-check-cache-370688148/abb62ab49b950e4b2ab5eb82c4386a9b199a08eb6fd93bdaccd0e4dbe10d29a2/data/53/53d242f8d6444bc2bd49e6689b4c7688fec8996c5e86863bf146a9eb28f54b5d", O_RDONLY|O_CLOEXEC) = 5
[pid 2634] 10:14:28 openat(AT_FDCWD, "/tmp/restic-check-cache-370688148/abb62ab49b950e4b2ab5eb82c4386a9b199a08eb6fd93bdaccd0e4dbe10d29a2/data/53/53d242f8d6444bc2bd49e6689b4c7688fec8996c5e86863bf146a9eb28f54b5d", O_RDONLY|O_CLOEXEC) = 5
[pid 2611] 10:14:28 openat(AT_FDCWD, "/tmp/restic-check-cache-370688148/abb62ab49b950e4b2ab5eb82c4386a9b199a08eb6fd93bdaccd0e4dbe10d29a2/data/53/53d242f8d6444bc2bd49e6689b4c7688fec8996c5e86863bf146a9eb28f54b5d", O_RDONLY|O_CLOEXEC) = 5
```

You might be able to recover somewhat by running restic rebuild-index and then running a new backup. If anything in the missing packs is still available in the location being backed up, a new backup will add it to the repository again. This might make your existing backups work again

I think I will try this next; if this restic check doesn't finish in the next 24 hours, I will SIGINT it and then run a restic rebuild-index. For the rebuild-index, should I use this PR? The restic-master head? Or something else?

Thanks again,
-- Durval.

Yeah, that's not looking promising. Likely the best thing to do is to do a restic rebuild-index, then run new backups of your three directories, and then forget all other snapshots. After that, you should be able to successfully do a restic prune.

I didn't touch the rebuild-index code, so you can do that with whatever version you want.

@cbane, just to keep you posted: I started a restic rebuild-index using your PR 5 days ago (I understand any version would do, but using yours make things simpler) and it has been running ever since. After the first few despairing days (when extrapolating from the percentages seemed to indicate it would run for over 30 days), it seems to have picked up some speed and should run for 'only' 10 more days or so (at least in its current 'counting files in repo' phase).

Presuming this rebuild-index finishes OK, the plan is then to run a restic prune with your PR. Will let you know how it goes.

Keeping everyone up-to-date: my free $300 GCloud credit ended, totally devoured by the 104GB VM I had to create to run the prune (and, I presume, also the rebuild-index). I'm running out of options :-/ Will update when/if I find a way out of this mess.

I tried the branch "prune-speedup" and results are very promising!

Backend: S3

# bin/restic version
restic 0.9.5 compiled with go1.12.4 on linux/amd64
# bin/restic prune
repository 6240cd5d opened successfully, password is correct
counting files in repo
building new index for repo
[1:30:34] 100.00%  319784 / 319784 packs
repository contains 319784 packs (5554019 blobs) with 1.445 TiB
processed 5554019 blobs: 0 duplicate blobs, 0 B duplicate
load all snapshots
find data that is still in use for 30 snapshots
[3:38:52] 100.00%  30 / 30 snapshots
found 5376708 of 5554019 data blobs still in use, removing 177311 blobs
will remove 0 invalid files
will delete 3526 packs and rewrite 4850 packs, this frees 26.925 GiB
[1:28:21] 100.00%  4850 / 4850 packs rewritten
counting files in repo
[1:20:27] 100.00%  314240 / 314240 packs
finding old index files
saved new indexes as [b7029105 797145b1 0ed8981e 7f19c455 dff4d9be a52d8e7a 0cf9f266 ab32a9e4 f34331b3 84c84cbb 563896c9 9902e35d 817ef96c 6b4dcfef 32d3e18c 1d782d96 b12554dd d4b69bcf 8ec94a43 66cbd035 8e9cb96d d74b5246 ca7d7ca3 1f209708 9fe26189 28414ee2 88ff07fb 24280466 8757a1f9 8706ff35 5fab942b 549409c3 f781a762 9417afec 4b2361aa 6b43f992 8da8dfe7 54ec498e 5be6fb8a 17411b83 270f3ce3 ba520d35 95913ad0 f8f15108 cbc67971 a419ff7c 875cfcc7 13f55ece bd488aa4 7f5b588a cddd40b4 7a79d1ce bd7e3d0f 2cdcd635 ee6e28c3 98146287 50867bde 41a056ae 836ce971 e9451c8b 0f9cc7e6 52dedc04 c4e8e7f6 2e4966f0 ba4fa5b3 ddc9a766 b995fd36 fd6b3ac8 1c12bcc3 4c98c3c9 39ac7cd5 42ebf4c1 1a48465e b5245192 73a73c5e daa6ee8d d26ce697 9f84d9b3 bc371def b141466a 6906b3c1 282ce115 d8024363 10f0b924 ad4fad64 9450aada 31378365 65d785b3 98b085d0 768f420c f22512b3 be3223ba 031d5488 f2b7fcf6 87177471 fd269664 b239b89e 6bf972ea 0d6f8f36 87362542 fff9c2cd 5c85ac76 f91daae1 dc594a83 220bdc83]
remove 1459 old index files
[2:33] 100.00%  8376 / 8376 packs deleted
done
# 

Now with dev version:

# bin/restic_linux_amd64 version
restic 0.9.5-dev (compiled manually) compiled with go1.12.4 on linux/amd64
# bin/restic_linux_amd64 prune
repository 6240cd5d opened successfully, password is correct
counting files in repo
building new index for repo
[57:21] 100.00%  314240 / 314240 packs
repository contains 314240 packs (5376708 blobs) with 1.419 TiB
processed 5376708 blobs: 0 duplicate blobs, 0 B duplicate
load all snapshots
find data that is still in use for 29 snapshots
[1:48:18] 100.00%  29 / 29 snapshots
found 5356653 of 5376708 data blobs still in use, removing 20055 blobs
will remove 0 invalid files
will delete 212 packs and rewrite 1427 packs, this frees 3.114 GiB
[14:16] 100.00%  1427 / 1427 packs rewritten
counting files in repo
[57:47] 100.00%  313618 / 313618 packs
finding old index files
saved new indexes as [004d6eb2 630de131 1b85faed 0fb7a978 bc500e05 34f7d187 447c3b59 ada047d2 5430430e 768f606e a5c341ea a75059c5 71d0fbec c63f5c56 ba43e69d f47699f5 d7cd2587 5826bcae 6250ec67 6af77aa4 cbf8c1f9 a7809b5f c3242748 8bb7d037 8ca69481 5a8877c3 fb30ea48 4bdb6269 eeeba466 7cdc444a bc15ddd5 c5544612 b8a5004c 2879403a c33778b7 e692013a 41ce8a1d 55b4be0a 5714b820 1adca569 b06ccd6b 16467da7 79ed066a 64c7e397 43217ede af7350a4 73c65a0f 35260990 a232ea42 c843bfa5 332aded7 0e294517 26984755 3c36a14b 68b2024e 267bf0b2 a41c4f64 aa46affb c7a6e2ac 0d34c60f 766d21f0 0d7b8b41 4fea4363 a1a3c5cd 73809a0e 56a67410 25314d47 a32ded24 68feae36 78ef5cbb b051a740 6a51f900 d2ee860f 1ad44787 c6c4358d 89de2f69 a157bd2b 77995b94 3bc58934 b905be42 0a1df2e7 ba67a98c 263b5266 7a809abc 34ff6f07 d96adc12 8f69fc74 ebb053eb 8fb68f6a 11224e7d 990f61f8 764941fc ed95513b 1c17c8aa 3226a7cb 76988c78 e4d8f156 b4d4c952 8c379d51 e968f3c9 f9cfff55 464cf3e1 f9d23c32 136957e3 02e397b1]
remove 105 old index files
[0:32] 100.00%  1639 / 1639 packs deleted
done
#

It looks like a big improvement! congrats!
I will post more results if I try on more machines in the upcoming weeks.

@fbarbeira From the output you posted, it looks like you're not actually using my improved version. Here's the output I get when I prune my big repository on Backblaze B2:

repository 399dfab1 opened successfully, password is correct
listing files in repo
loading index for repo
[0:02] 100.00%  71 / 71 index files
checking for packs not in index
repository contains 208840 packs (1675252 blobs) with 1006.203 GiB
processed 1675252 blobs: 0 duplicate blobs, 0 B duplicate
load all snapshots
find data that is still in use for 32 snapshots
[0:16] 100.00%  32 / 32 snapshots
found 1675113 of 1675252 data blobs still in use, removing 139 blobs
will remove 0 invalid files
will delete 4 packs and rewrite 24 packs, this frees 26.404 MiB
[3:31] 100.00%  24 / 24 packs rewritten
saving new index
[10:31] 70 index files
remove 71 old index files
[0:04] 100.00%  28 / 28 packs deleted
done

The main source of slowness in that is the limited upload speed over my cable modem.

The output from restic version should also look something like this:

restic 0.9.5 (v0.9.5-31-g09b92d6d) compiled with go1.11.6 on linux/amd64

Any intentions to merge this update @fd0 ?

I would very much appreciate this improvement making it into an official release as well. Backup rotation has become so slow that restic is starting to not really be an option anymore, sadly.

@cbane, one item that is definitely not a blocker but figured I'd flag: prune pack rewrites get slower as repos grow.

For example, the pack rewrite step for a 3,984,097 pack repo rewrites packs at 110 packs/second on a i3.8xlarge talking to an S3 bucket in the same AZ.

The same instance + backing store combo with a smaller repo (450,473) rewrites at 200 packs/second.

@pmkane, is that speed disparity there even if they're rewriting the same number of packs? Or is it always there no matter how many packs are being rewritten? Also, is it just the pack rewriting, or do other stages slow down as well?

There's nothing obvious in the code that would slow it down as the repository gets bigger. I've added profiling support to the pack rewrite stage in my local version to help track down the source of the slowness. However, my largest repository is only about 200,000 packs, which doesn't give me a good comparison for what you're seeing. Would you be willing to run it with profiling enabled and make the output files available?

@cbane, not sure if it's a function of number of packs or repo size. I can run some tests on a duplicate of our smaller repo and see. Happy to run a version with profiling and share the results.

Some sample timings for the 460k pack repo -- 3.7mm timings follow:

loading index for repo
[0:01] 100.00% 154 / 154 index files

find data that is still in use for 36 snapshots
[0:34] 100.00% 36 / 36 snapshots
[0:26] 100.00% 4555 / 4555 packs rewritten (175 packs/second)
[0:21] 151 index files
[0:01] 100.00% 11401 / 11401 packs deleted

3,752,505 pack repo. It's also worth noting that memory usage balloons from ~1GB RSS to 8GB RSS in the "find data that is still in use for 14 snapshots" step. Restic eventually ends up at ~16GB RSS. Maybe unavoidable, given the large repo size:

loading index for repo
[0:33] 100.00% 1188 / 1188 index files

find data that is still in use for 14 snapshots
[2:12] 100.00% 14 / 14 snapshots
[49:07] 100.00% 339187 / 339187 packs rewritten (115 packs/second)
saving new index
[10:59] 1176 index files
remove 1188 old index files
[4:51] 100.00% 409728 / 409728 packs deleted

@cbane, I apologize. Red herring on the prune speeds -- when I was doing some independent profiling I spotted that the two repos were in different AZs, so the difference was entirely due to differing latency to the more distant AZ.

In our large repo (20.791TiB, 40,522,693 blobs), we are getting the following error while pruning in the "find data that is still in use" step:

tree 5e40b6de93549df6443f55f0f68f24bf36671e28590579c78bc62f7557490c56 not found in repository
github.com/restic/restic/internal/repository.(Repository).LoadTree
internal/repository/repository.go:711
github.com/restic/restic/internal/restic.FindUsedBlobs.func3
internal/restic/find.go:52
github.com/restic/restic/internal/restic.FindUsedBlobs.func3
internal/restic/find.go:74
github.com/restic/restic/internal/restic.FindUsedBlobs.func4
internal/restic/find.go:89
gopkg.in/tomb%2ev2.(Tomb).run
vendor/gopkg.in/tomb.v2/tomb.go:163
runtime.goexit
/usr/lib/golang/src/runtime/asm_amd64.s:1337

all of the backups completed successfully and a restic check doesn't return any errors.

Any additional digging worth doing here, @cbane ?

A rebuild-index allowed the prune to succeed. Not sure if there's a good way to make prune more resilient, so it could take care of the issue natively.

Hmm, got the same error again today. Resolved with a rebuild-index, but am starting to wonder if the prune itself might be causing the problem. A restic check comes back clean each time.

tree 7dc9112b587a2b67a2459e6badf7c14f986408e05dbe8a68e7458a30740ea951 not found in repository
github.com/restic/restic/internal/repository.(Repository).LoadTree
internal/repository/repository.go:711
github.com/restic/restic/internal/restic.FindUsedBlobs.func3
internal/restic/find.go:52
github.com/restic/restic/internal/restic.FindUsedBlobs.func3
internal/restic/find.go:74
github.com/restic/restic/internal/restic.FindUsedBlobs.func4
internal/restic/find.go:89
gopkg.in/tomb%2ev2.(Tomb).run
vendor/gopkg.in/tomb.v2/tomb.go:163
runtime.goexit
/usr/lib/golang/src/runtime/asm_amd64.s:1337

We do once a week full restores to verify backup integrity, to complement daily spot file restores.

While restic check did not show any issues with the backup, full restores fail due to a missing blob. So it does look like something may have gotten pruned along the way that shouldn't have.

Unclear if it's a bug in the speedup branch, something in the base prune code or something else entirely. I don't have a great reproducible test case, sadly, but this is the second time we have seen this type of repo corruption with our largest repo. Our smaller repos have not shown any issues.

Unfortunately testing with stock prune is impossible, due to the size of the repos.

We're going to revert to EBS snapshotting, for now, but will continue to monitor this issue and others and are happy to test where it makes sense!

I added a small PR #2507 which should improve the situation. I would appreciate if some of you could run some tests on this...
Thanks!

Reading through the prune code - repacking reads blobs and then saves new packs serially. If you're repacking over the network that's going to be the bottleneck.

@DurvalMenezes is your NAS on the local network or over the internet? If on the local network do you connect to it via wifi or ethernet?

It seems like an easy win would be to parallelize reading blobs / saving packs to the network.

Separately, I wonder if a better strategy would be to try and make prune incremental instead. Something like duplicacy's two step fossil collection: https://github.com/gilbertchen/duplicacy/wiki/Lock-Free-Deduplication#two-step-fossil-collection

Separately, I wonder if a better strategy would be to try and make prune incremental instead. Something like duplicacy's two step fossil collection: https://github.com/gilbertchen/duplicacy/wiki/Lock-Free-Deduplication#two-step-fossil-collection

Please see #1141 for a adiscussion on this topic.

The two new commands 'cleanup-index' and 'cleanup-packs' from PR #2513 should also improve the situation a lot. With these commands you can do a prune without repacking if your repo is sane..

So I accidentally did two things, I ran an hourly backup 11 times an hour instead of 1, and I didn't run my forget job for the last 384 days or so. I have 101417 snapshots. I figured it would be too slow to forget/prune this, I think I'm just going to delete it.

You may try out #2718 - however, there is no improvement for "forgetting" snapshots.
If the forget step is your trouble I would advice you to:

• figure out which snapshots you want to keep, e.g. by looking in your last backup logs or just make another backup.
• delete all but those files manually from your repository's /snapshots dir
• after that, run prune (if you want with #2718)

I only meant to keep them for a few days, so I'll just delete the entire directory on S3 and start over. I started the forget process and it was taking a very long time, I thought either the newer branch would help (doesn't look like it), or at the very least someone might find it amusing.

I really want to try out this branch (or really, wish it was pulled into master). I tried to test and get

Fatal: unable to open config file: Stat: The AWS Access Key Id you provided does not exist in our records.

Works without issue using the master build. I also was able to use the branch at https://github.com/restic/restic/pull/2340 without issue.

I am able to use this branch on a NFS mounted repository, only the AWS stored repo is non-functional. I have no idea how to troubleshoot...

thanks for all the good work regardless

@vtwaldo21 Which branch did you try out? Was it #2718?
You error message indicates you have some trouble with your AWS credentials. This may also explain why NFS did work without a problem.

Do other restic commands work with you AWS repository?

@aawsome , I'm an idiot and transposed the branch/issue numbers and really confused things. sorry.

Branch 2718 worked just fine (on both AWS and NFS backed repos). I can't say definitively it was faster or not as it is still running
Branch 2340, was the one with the issue and why I was in this forum thread.

Branch 2340 is 0.9.5 based so a little older but not too bad. I was doing simple tests like this:

restic binary is path is just installed via RPM (0.9.6)

restic snapshots
[works]
restic.2718/restic snapshots
[works]
restic.2340/restic snapshots
Fatal: unable to open config file: Stat: The AWS Access Key Id you provided does not exist in our records.

So while the error implies something with AWS credentials, i'm literally running the commands back to back to back with no variable changes. It seems there is something else wrong with this branch.

My prunes take days for a ~2TB repo, so very interested in both 2718 and 2340 being merged into master

@cbane thank you very much for all your work! We've just merged #2718, which reworks the prune code, and I think it resolves this issue as well as #2340.

I'm very sorry at how we handled your contribution, I hope you can accept my apologies. :disappointed:

I've talked with @MichaelEischer, he mentioned that the PR #2340 includes more idea that haven't been implemented yet, so I'm reopening this issue.

@cbane are you interested in working with us to get these ideas merged into the current code? I can totally understand if not, then we'll go through your PR and extract them ourselves :)

We also had a short discussion yesterday and tried to identify what may have gone wrong, so we can do better next time. Some points we identified were:

• The PR changed a highly sensitive area in the code (prune, which ultimately removes data), so it needs extra scrutiny
• @MichaelEischer as well as @aawsome tried to review the PR and said it was too large
• The PR was just two commits, one of them replaced all the code with other, completely different code, this is very hard to review
• The PR included at least 4-5 different ideas and improvements, which makes it even harder to review

Any other things I missed?

The changes to cmd_prune.go from #2340 are as far as I can see completely superseded by #2718 and #2842 by @aawsome . #3006 also replaces the index/index.go changes.

I've extracted the highly parallelized tree walking implementation from checker.go (see https://github.com/MichaelEischer/restic/tree/streamtrees), that allows implementing a parallel FindUsedBlobs with a few additional lines of code. It is also reusable for the copy command. My mentioned branch also unifies the code used to parallelize index and snapshot loading. I'll split the branch into smaller PRs.

That seems to leave the usage of the backend connection count and GOMAXPROCS to automatically adjust to IO/CPU parallelism as remaining part of #2340.

I've noticed that neither #2340 nor the other prune PR currently parallelize the upload of the rewritten index.