<p>Restic修剪:内存不足</p>

创建于 2018-04-16  ·  31评论  ·  资料来源: restic/restic

restic version

Restic 0.8.3(v0.8.3-0-g272ccec7)
在Linux / amd64上使用go1.10编译

您是如何精确地休息的?

RESTIC_PASSWORD = foobar
RESTIC_REPOSITORY = /数据/乔/ RESTIC
休息时间修剪

您使用什么后端/服务器/服务来存储存储库?

直接文件访问。

预期行为

Restic修剪回购协议。

实际行为

2018-04-16 18:33:51: Starting restic prune
counting files in repo
building new index for repo
[11:59] 100.00%  291017 / 291017 packs

incomplete pack file (will be removed): 02b612398378d4b6d7b6a77b5f51cf3c541baa78f542886810c27488fc52f8a2
incomplete pack file (will be removed): 08ce4b79b2adacd1d85373c59ad28777e215cec52b16bf9963cb03106e1d547f
incomplete pack file (will be removed): 08d76e1953f8b06d936f46679986f0af9bfb0260ca1b81e641d30262f4e2fb69
incomplete pack file (will be removed): 0ada7a6774140391104de82da908a1027eaedc97be83a2c30ae26db85093aba1
incomplete pack file (will be removed): 0f960d42718f0e0b29ea10a47fef530638811fd1620843e13e4c4c3c139c9477
incomplete pack file (will be removed): 11aa5b87e3a44ff1678de573e21280ff56e8cdf93755c363c7ba1a03537b84b0
incomplete pack file (will be removed): 13b211c7fc26e7de47e8a5387749bb93f7737217809143e01caf7f826deeea13
incomplete pack file (will be removed): 13f00bb9960f95b0d71abc9ff5e5b86f53d370cd602a4c45e03505cefab328ab
incomplete pack file (will be removed): 1b0be1e761e0c369aaed7b2d03f07163854946dfe7b6bb749d4ff5d5c62cfd61
incomplete pack file (will be removed): 1ee341e8778661a76bc0f4d2a285457851cb5984f6ac6ee51d42bbdf705025ab
incomplete pack file (will be removed): 26de4ab07458e880217accac26098edcc46622d11d20dcbb57996cfebedbc859
incomplete pack file (will be removed): 274ede1a69f1f2634abeb6f7f7a9af2a4027a7f224d7a61a6e209ef8ee4c152f
incomplete pack file (will be removed): 383bad67ef592da43d0cc9793bc6e1afc4a4327fc2c504b753111888b6c189f3
incomplete pack file (will be removed): 39d74d322db6b80f613efabe4c1b7242b3ff31ab936b60a0a4ee903713f9b299
incomplete pack file (will be removed): 419b5a6e3ebbab53150b5106316e8f3183ef7dd6d3b0c10de0a611dedb7ea846
incomplete pack file (will be removed): 44ba8b9892a33391201339cbea4c963707ee16d30c6ca96d38d96bbeadfe23ca
incomplete pack file (will be removed): 45b75bb12def8334143681bf89811c68baace1376311e3ca79478021faf6b062
incomplete pack file (will be removed): 47947ddb8dd54856e22ec44b0de268668b89ccac77e6145a61f05870ab23c57d
incomplete pack file (will be removed): 4ba9ef53b1579d10d0682153382a6a30d12e444bf3ac0b55dd3e5bbe1f151d9f
incomplete pack file (will be removed): 4d36946b6c368baf824590d0846b1c28cb34943c8fd16734b83847b42ded8dab
incomplete pack file (will be removed): 61b294bea904324ef980d75c5b5da2f0e8b6d9be1aa16e55b7606afe5623df69
incomplete pack file (will be removed): 65555bcda953ddb00af9e3d141a4c274614d5dfd3a6345bf396872028852797c
incomplete pack file (will be removed): 716d9a5a48c2d364069fc9c9576c1683bd99cecc7b0232c53034a70bca9aed2b
incomplete pack file (will be removed): 71766e7583ba835480aa87e02566b23c26ba940ecc0f697beb2aaa0621422f4c
incomplete pack file (will be removed): 74f82d71b0f881c95f5eb54668355ce2d9f32a5a6087ec61e52618f909af377c
incomplete pack file (will be removed): 8476543253237a28264c87c70f41eb87883488f8d84314045cc07c3e7deee656
incomplete pack file (will be removed): 8d9d77e05a32a84da1a62159c245a004d55386923137f406c74b1b01281d07d3
incomplete pack file (will be removed): 8fa209c571b2cace97f5725c12c3a110deef8b8f7cfe6705f28175ac6182e9b8
incomplete pack file (will be removed): 9146ec8c10a77c248aa0db6aa291c25b83498848afddfbb4bb33b9928b003fdf
incomplete pack file (will be removed): 9f5e6c422683e05fe9cb7212d44c6492c55d24cb55230407cc59ebd06c96dde7
incomplete pack file (will be removed): a4ac5898b871d3a4343e2fd4c88aaae2e1a31879cce0d90d4e93527334a9deeb
incomplete pack file (will be removed): aa0d2b035ebcab381cd816ba2000d61c43f824621fafc74b81e5cfa1524ae94c
incomplete pack file (will be removed): aa2f5ba3a21a7b12ed23ab198ee457c5fde4921344cd350f2cb9ea675d05a1d6
incomplete pack file (will be removed): aa8f152200a3657728e68eddc1a9ad4c3cecf20af568d4fa01727ffb86005856
incomplete pack file (will be removed): af13847a8a95013afc34b59dfb40daa1c74f0067100a49adf5b1279559d3afd5
incomplete pack file (will be removed): af287ee5aa3c436cba495f4a73f30bf43d23e995d94913c9e64e12dd670b0a36
incomplete pack file (will be removed): ba38890edd8e2c84feb3af506427b6fd5cb4bd4150d7a5784f15f82e01598ca1
incomplete pack file (will be removed): bca933d271cd67feb9b42c55eb168d9f44bb568a121d3947fa443975e47e1ba1
incomplete pack file (will be removed): bd6cdb11204f445cc413cd04fbf56038eee417bc6312dd26624c037eba153417
incomplete pack file (will be removed): c436f5f332c6dbefa57456c8b0532cb3a89beb99564ab67b3ecc433bb0654df3
incomplete pack file (will be removed): cffa984794599843ec0b673c906a84f7813a4ee2346af3bacb6b7c2a217e172f
incomplete pack file (will be removed): d1aeb4847d269c3e6c469e2ac7a82e846f7891f357b0c3b25ef7fa8cc1bd9388
incomplete pack file (will be removed): d2269e9659755adb4d0c8944cb2abe1d6b2e75fa48e9c2ac749ec07c1222e196
incomplete pack file (will be removed): d29e8e66df6380d8f6bf65e18f35bc4597539e73c9a1932bca3dc067fa219ad6
incomplete pack file (will be removed): eb9e4049591e77ad69bc52d9bf830872798bc5c0814ad7cf65eb03befc38bed8
incomplete pack file (will be removed): ec458035798f3e4159314a50e0b7e3f5b8766564aef307c7f10fa7ac3f6dfa5d
incomplete pack file (will be removed): ecf1542bf4a15bc44d447973fc6312fe1a4e2528b5fd07aa9264f5b4bae6820c
incomplete pack file (will be removed): ee2ce3bd4a13ee395ad3a4a95dd0aa017e0b17ecf2cbd1faa66124e510717762
incomplete pack file (will be removed): ee637a1a84fc5819dee56c3192f38bf3895a00b42f6d3ddec5e2ed43c208e18a
incomplete pack file (will be removed): ef4e136efdd73d6e667c8afc8deccaf1d8e6c425d66eb377d5fa1fa85c484394
incomplete pack file (will be removed): f1d0b28ff0c5bdb8d77f9a7381324a54f574a7e9eb0f7e7320f315a26a71cb7a
repository contains 290966 packs (4042496 blobs) with 1.350 TiB
processed 4042496 blobs: 14417 duplicate blobs, 947.112 MiB duplicate
load all snapshots
find data that is still in use for 30 snapshots
[14:50] 100.00%  30 / 30 snapshots

found 4020862 of 4042496 data blobs still in use, removing 21634 blobs
will remove 51 invalid files
will delete 110 packs and rewrite 34143 packs, this frees 1.797 GiB
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0xae011d, 0x16)
    /usr/local/go/src/runtime/panic.go:619 +0x81
runtime.sysMap(0xc4f7c60000, 0x490000, 0x43fd00, 0xe91e58)
    /usr/local/go/src/runtime/mem_linux.go:216 +0x20a
runtime.(*mheap).sysAlloc(0xe78660, 0x490000, 0x7fd3f522c098)
    /usr/local/go/src/runtime/malloc.go:470 +0xd4
runtime.(*mheap).grow(0xe78660, 0x248, 0x0)
    /usr/local/go/src/runtime/mheap.go:907 +0x60
runtime.(*mheap).allocSpanLocked(0xe78660, 0x248, 0xe91e68, 0xc45cd39ee0)
    /usr/local/go/src/runtime/mheap.go:820 +0x301
runtime.(*mheap).alloc_m(0xe78660, 0x248, 0x410101, 0xc41e8c7fff)
    /usr/local/go/src/runtime/mheap.go:686 +0x118
runtime.(*mheap).alloc.func1()
    /usr/local/go/src/runtime/mheap.go:753 +0x4d
runtime.(*mheap).alloc(0xe78660, 0x248, 0xc45c010101, 0x4142ac)
    /usr/local/go/src/runtime/mheap.go:752 +0x8a
runtime.largeAlloc(0x48f87c, 0x450101, 0x7fd3f5ca56c8)
    /usr/local/go/src/runtime/malloc.go:826 +0x94
runtime.mallocgc.func1()
    /usr/local/go/src/runtime/malloc.go:721 +0x46
runtime.systemstack(0x0)
    /usr/local/go/src/runtime/asm_amd64.s:409 +0x79
runtime.mstart()
    /usr/local/go/src/runtime/proc.go:1170

goroutine 1 [running]:
runtime.systemstack_switch()
    /usr/local/go/src/runtime/asm_amd64.s:363 fp=0xc480ad81c8 sp=0xc480ad81c0 pc=0x454010
runtime.mallocgc(0x48f87c, 0x9cfd20, 0x979301, 0xc4d6371720)
    /usr/local/go/src/runtime/malloc.go:720 +0x8a2 fp=0xc480ad8268 sp=0xc480ad81c8 pc=0x410882
runtime.makeslice(0x9cfd20, 0x48f87c, 0x48f87c, 0x1, 0x1, 0x58f)
    /usr/local/go/src/runtime/slice.go:61 +0x77 fp=0xc480ad8298 sp=0xc480ad8268 pc=0x43fdc7
github.com/restic/restic/internal/repository.Repack(0xb5a640, 0xc420068e40, 0xb62260, 0xc4202510a0, 0xc424eb4120, 0xc475dfecf0, 0xc4200c38c0, 0x9c5582208afec32f, 0xc475dfecf0, 0xc475dfed20)
    /tmp/restic-build-762914258/src/github.com/restic/restic/internal/repository/repack.go:60 +0xec3 fp=0xc480ad8a60 sp=0xc480ad8298 pc=0x752d93
main.pruneRepository(0xc420024012, 0x10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    src/github.com/restic/restic/cmd/restic/cmd_prune.go:278 +0x1e99 fp=0xc480ad9990 sp=0xc480ad8a60 pc=0x94d959
main.runPrune(0xc420024012, 0x10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    src/github.com/restic/restic/cmd/restic/cmd_prune.go:85 +0x153 fp=0xc480ad9ab8 sp=0xc480ad9990 pc=0x94b943
main.glob..func17(0xe6b180, 0xe8ffd8, 0x0, 0x0, 0x0, 0x0)
    src/github.com/restic/restic/cmd/restic/cmd_prune.go:25 +0x4d fp=0xc480ad9bb8 sp=0xc480ad9ab8 pc=0x95e07d
github.com/restic/restic/vendor/github.com/spf13/cobra.(*Command).execute(0xe6b180, 0xe8ffd8, 0x0, 0x0, 0xe6b180, 0xe8ffd8)
    /tmp/restic-build-762914258/src/github.com/restic/restic/vendor/github.com/spf13/cobra/command.go:698 +0x46d fp=0xc480ad9c60 sp=0xc480ad9bb8 pc=0x6b34ed
github.com/restic/restic/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xe69e60, 0xffffffffffffffff, 0x0, 0x19)
    /tmp/restic-build-762914258/src/github.com/restic/restic/vendor/github.com/spf13/cobra/command.go:783 +0x2e4 fp=0xc480ad9d90 sp=0xc480ad9c60 pc=0x6b3c64
github.com/restic/restic/vendor/github.com/spf13/cobra.(*Command).Execute(0xe69e60, 0x24, 0xc420217eb8)
    /tmp/restic-build-762914258/src/github.com/restic/restic/vendor/github.com/spf13/cobra/command.go:736 +0x2b fp=0xc480ad9dc0 sp=0xc480ad9d90 pc=0x6b395b
main.main()
    src/github.com/restic/restic/cmd/restic/main.go:69 +0x1d6 fp=0xc480ad9f88 sp=0xc480ad9dc0 pc=0x95c796
runtime.main()
    /usr/local/go/src/runtime/proc.go:198 +0x212 fp=0xc480ad9fe0 sp=0xc480ad9f88 pc=0x42b472
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc480ad9fe8 sp=0xc480ad9fe0 pc=0x456b31

goroutine 5 [syscall, 28 minutes]:
os/signal.signal_recv(0x0)
    /usr/local/go/src/runtime/sigqueue.go:139 +0xa6
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 6 [chan receive, 28 minutes]:
github.com/restic/restic/internal/restic.init.0.func1.1()
    /tmp/restic-build-762914258/src/github.com/restic/restic/internal/restic/lock.go:266 +0xef
created by github.com/restic/restic/internal/restic.init.0.func1
    /tmp/restic-build-762914258/src/github.com/restic/restic/internal/restic/lock.go:263 +0x35

goroutine 7 [select, 28 minutes, locked to thread]:
runtime.gopark(0xb01d80, 0x0, 0xad384f, 0x6, 0x18, 0x1)
    /usr/local/go/src/runtime/proc.go:291 +0x11a
runtime.selectgo(0xc420042f50, 0xc42002c2a0)
    /usr/local/go/src/runtime/select.go:392 +0xe50
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal_unix.go:549 +0x1f4
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1

goroutine 17 [chan receive, 28 minutes]:
github.com/restic/restic/internal/restic.init.2.func1(0xc42002c1e0)
    /tmp/restic-build-762914258/src/github.com/restic/restic/internal/restic/progress_unix.go:17 +0xa3
created by github.com/restic/restic/internal/restic.init.2
    /tmp/restic-build-762914258/src/github.com/restic/restic/internal/restic/progress_unix.go:16 +0xab

goroutine 18 [chan receive, 28 minutes]:
main.CleanupHandler(0xc42021c1e0)
    src/github.com/restic/restic/cmd/restic/cleanup.go:62 +0x5d
created by main.init.0
    src/github.com/restic/restic/cmd/restic/cleanup.go:24 +0x6d

goroutine 33 [select, 3 minutes]:
main.refreshLocks(0xe71be0, 0xc4201322a0)
    src/github.com/restic/restic/cmd/restic/lock.go:72 +0x221
created by main.lockRepository
    src/github.com/restic/restic/cmd/restic/lock.go:49 +0x373

goroutine 40 [select]:
github.com/restic/restic/internal/restic.(*Progress).reporter(0xc4200c38c0)
    /tmp/restic-build-762914258/src/github.com/restic/restic/internal/restic/progress.go:169 +0x130
created by github.com/restic/restic/internal/restic.(*Progress).Start
    /tmp/restic-build-762914258/src/github.com/restic/restic/internal/restic/progress.go:96 +0x132
2018-04-16 19:02:18: Terminated: restic prune (return code 2)

重现行为的步骤

获取具有4 GiB内存和Intel(R)内核的Debian系统(Linux备份4.9.0-6-amd64#1 SMP Debian 4.9.82-1 + deb9u3(2018-03-02)x86_64 GNU / Linux) TM)i3-3225 CPU @ 3.30GHz。 然后创建一个大小约为1.5 TB的存储库,并在服务器端运行Restic修剪。

您是否知道可能是什么原因造成的?

我在其中运行restic的系统具有4 GiB的内存。 当然,更多会更好。 但是,如果Restic不会因此类错误而失败,那就更好了。

您有解决该问题的想法吗?

当然。 解决方案是向系统添加更多内存。 或提高Restic的效率(并确保没有资源泄漏)。

Restic对您有帮助还是让您开心呢?

钓鱼赞美,是吗? 当然可以,为什么不呢。 毫无疑问,Restic很酷。 我喜欢客户端加密,正确使用KDF,支持ACL,通过FUSE挂载备份。 它击败了我以前的rsync方法。 但这是否可以减少资源的浪费? :-)

prune bug

最有用的评论

嗨,亚历山大,我在这里没有任何技术要补充,但是在您发表最后的评论后,我觉得我必须这样说:在与您进行的所有交流中,您除了专业,友善而且反应迅速。 我之前曾在这里报道过虚假的错误(例如,事实根源于硬件问题),并且您一直超级支持。 任何开放源代码项目都可以让您成为他们的赞助人,这很幸运,它清楚地表明您正在超越极限。 这是一个了不起的工具,我们都知道。 请放心,即使我相信您,当您昨天说自己心情不好时,也肯定不会在您的发言中出现。 继续进行出色的工作,并再次感谢您继续不懈地改进Restic。 约翰内斯,干杯

所有31条评论

回顾回溯,我想核心问题在于goroutine 1:

https://github.com/restic/restic/blob/6a34e0d10fb978cb8988a460bf4b04e9976cca17/internal/repository/repack.go#L60

这就提出了一个问题:我们可以更有效地重新包装吗? 例如,在顺序批量操作中加载blob或使用写时复制来维护blob的数据……需要比我更多的架构知识。

是的,修剪需要太多的内存,因此效率更高。 我只需要花一些时间对其进行优化。 感谢您的举报!

不确定分享此内容的最佳地点,但可以在此处添加一个数据点。 昨晚引起一些OOM错误后,我再次禁用了Restic。 在我们的服务器上进行备份时,它达到了使用约25GB RAM的位置:

Aug 21 03:24:35 courtlistener kernel: [3131661.383314] Out of memory: Kill process 28903 (restic) score 359 or sacrifice child
Aug 21 03:24:35 courtlistener kernel: [3131661.384564] Killed process 28903 (restic) total-vm:25856956kB, anon-rss:24415312kB, file-rss:0kB
Aug 21 03:24:35 courtlistener kernel: [3131661.400102] postgres invoked oom-killer: gfp_mask=0x2000d0, order=2, oom_score_adj=0

在具有大约20GB RAM的数据库的服务器上,我们保留了诸如Restic之类的开销,但是我们无法仅保留30GB RAM(或更多?)来进行备份。 我们仍在使用restic来流式传输数据库备份,但是,这实在是太多了。

嗯,这是在备份期间? 还是修剪?

备份期间。 我们已经禁用了修剪功能。 没有修剪只是意味着Backblaze向我们收取更多费用,因此在性能/内存问题解决之前将其禁用即可。 但是现在我们也不得不禁用备份。

抱歉,对此感到抱歉。

好吧,这就是它! 我们和任何人都知道,扩展并非易事,有时我们会推波助澜。 总的来说,我还是喜欢Restic。 流式备份? 嗯吻手指。 美味的。

@mlissner您的存储库大小是多少?

昨天我心情不太好,但今天好多了:)

嗨,亚历山大,我在这里没有任何技术要补充,但是在您发表最后的评论后,我觉得我必须这样说:在与您进行的所有交流中,您除了专业,友善而且反应迅速。 我之前曾在这里报道过虚假的错误(例如,事实根源于硬件问题),并且您一直超级支持。 任何开放源代码项目都可以让您成为他们的赞助人,这很幸运,它清楚地表明您正在超越极限。 这是一个了不起的工具,我们都知道。 请放心,即使我相信您,当您昨天说自己心情不好时,也肯定不会在您的发言中出现。 继续进行出色的工作,并再次感谢您继续不懈地改进Restic。 约翰内斯,干杯

我遇到的问题与#1830中的Restic检查相同。
尝试将GOGC设置为20,但没有运气。 运行一段时间(8 GB内存,〜1,5 TB回购)后,restic检查仍然无法通过操作。

在/ tmp / restic-check-cache-961467159中使用临时缓存
存储库02a14a0b已成功打开,密码正确
为存储库创建排他锁
负荷指标
检查所有包
检查快照,树木和斑点
被杀


杀死过程16350(restic)总-VM:2428260kB,anon- RSS:2375472kB ,文件级RSS:0KB ,shmem- RSS:0KB

@mlissner您的存储库大小是多少?

抱歉,在这种情况下,我不确定“存储库”的含义是什么,但是我猜您是说我们处于备受推崇的状态吗? 如果正确,那么存储桶中的大小约为5.2TB。

如果您是说我们要备份多少,那么它大约有2TB的文件,可能有1500万个文件? 大多数情况都不会改变,但是我们每天都会得到更多。 我经营法律文件档案库CourtListener.com。

@ johndoe31415感谢您的客气话! 有时仍有大量工作要做...

我有同样的问题(8GB RAM,回购是267GB)

$ restic version
restic 0.9.3 compiled with go1.10.4 on linux/amd64

命令: restic -r sftp:[email protected]:/mybackup --password-file mypassword --cache-dir /var/cache/restic forget --prune --keep-daily=7 --keep-weekly=4 --keep-monthly=6 --keep-yearly=1

1 snapshots have been removed, running prune
counting files in repo
building new index for repo
[1:14:56] 100.00%  56025 / 56025 packs

repository contains 56025 packs (4844700 blobs) with 257.876 GiB
processed 4844700 blobs: 0 duplicate blobs, 0B duplicate
load all snapshots
find data that is still in use for 44 snapshots
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0xb92259, 0x16)
        /usr/lib/go-1.10/src/runtime/panic.go:616 +0x81
runtime.sysMap(0xc50f780000, 0x12980000, 0xf6ce00, 0xf85fd8)
        /usr/lib/go-1.10/src/runtime/mem_linux.go:216 +0x20a
runtime.(*mheap).sysAlloc(0xf6c6e0, 0x12980000, 0x7f7d482609c8)
        /usr/lib/go-1.10/src/runtime/malloc.go:470 +0xd4
runtime.(*mheap).grow(0xf6c6e0, 0x94c0, 0x0)
        /usr/lib/go-1.10/src/runtime/mheap.go:907 +0x60
runtime.(*mheap).allocSpanLocked(0xf6c6e0, 0x94c0, 0xf85fe8, 0x7f7d6bffed88)
        /usr/lib/go-1.10/src/runtime/mheap.go:820 +0x301
runtime.(*mheap).alloc_m(0xf6c6e0, 0x94c0, 0x410101, 0xc41a96e1ff)
        /usr/lib/go-1.10/src/runtime/mheap.go:686 +0x118
runtime.(*mheap).alloc.func1()
        /usr/lib/go-1.10/src/runtime/mheap.go:753 +0x4d
runtime.(*mheap).alloc(0xf6c6e0, 0x94c0, 0x7f7d6b010101, 0x415afc)
        /usr/lib/go-1.10/src/runtime/mheap.go:752 +0x8a
runtime.largeAlloc(0x12980000, 0x450101, 0x7f7d79aded90)
        /usr/lib/go-1.10/src/runtime/malloc.go:826 +0x94
runtime.mallocgc.func1()
        /usr/lib/go-1.10/src/runtime/malloc.go:721 +0x46
runtime.systemstack(0xc400000000)
        /usr/lib/go-1.10/src/runtime/asm_amd64.s:409 +0x79
runtime.mstart()
        /usr/lib/go-1.10/src/runtime/proc.go:1175

goroutine 1 [running]:
runtime.systemstack_switch()
        /usr/lib/go-1.10/src/runtime/asm_amd64.s:363 fp=0xc4d2cc6b88 sp=0xc4d2cc6b80 pc=0x4552d0
runtime.mallocgc(0x12980000, 0xaffe40, 0xbb6b01, 0xaed4a0)
        /usr/lib/go-1.10/src/runtime/malloc.go:720 +0x8a2 fp=0xc4d2cc6c28 sp=0xc4d2cc6b88 pc=0x4120d2
runtime.newarray(0xaffe40, 0x110000, 0xc42013c000)
        /usr/lib/go-1.10/src/runtime/malloc.go:855 +0x6a fp=0xc4d2cc6c58 sp=0xc4d2cc6c28 pc=0x41245a
runtime.makeBucketArray(0xb32640, 0x42f5cad737fd4814, 0xc4d2cc6cd0, 0x401e32)
        /usr/lib/go-1.10/src/runtime/hashmap.go:881 +0xe2 fp=0xc4d2cc6c90 sp=0xc4d2cc6c58 pc=0x40a6b2
runtime.hashGrow(0xb32640, 0xc44ee1c960)
        /usr/lib/go-1.10/src/runtime/hashmap.go:905 +0x80 fp=0xc4d2cc6ce0 sp=0xc4d2cc6c90 pc=0x40a830
runtime.mapassign(0xb32640, 0xc44ee1c960, 0xc4d2cc6f9f, 0xc4cd1f7d40)
        /usr/lib/go-1.10/src/runtime/hashmap.go:579 +0x2b7 fp=0xc4d2cc6d70 sp=0xc4d2cc6ce0 pc=0x409777
github.com/restic/restic/internal/restic.BlobSet.Insert(...)
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/internal/restic/blob_set.go:26
github.com/restic/restic/internal/restic.FindUsedBlobs(0xc15640, 0xc42020e9c0, 0xc1d760, 0xc42021fa40, 0x467d86ab22e4ce70, 0x21a44382a1438885, 0x829c686ff3f4eb07, 0x2bbf9f1c26cb983a,
0xc44ee1c960, 0xc44ee1c990, ...)
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/internal/restic/find.go:9 +0x10c fp=0xc4d2cc6fe8 sp=0xc4d2cc6d70 pc=0x6f9ecc
github.com/restic/restic/internal/restic.FindUsedBlobs(0xc15640, 0xc42020e9c0, 0xc1d760, 0xc42021fa40, 0xad0966088f423528, 0xc0855b2604177a2c, 0x416e998bdcc27c99, 0xd45bd561fbc152b3,
0xc44ee1c960, 0xc44ee1c990, ...)
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/internal/restic/find.go:31 +0x5a6 fp=0xc4d2cc7260 sp=0xc4d2cc6fe8 pc=0x6fa366
github.com/restic/restic/internal/restic.FindUsedBlobs(0xc15640, 0xc42020e9c0, 0xc1d760, 0xc42021fa40, 0x7ed337032395c12e, 0x697a1ab6f4be111b, 0x77f9fa7adfc1bd09, 0x1821c35434fcefb9,
0xc44ee1c960, 0xc44ee1c990, ...)
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/internal/restic/find.go:31 +0x5a6 fp=0xc4d2cc74d8 sp=0xc4d2cc7260 pc=0x6fa366
github.com/restic/restic/internal/restic.FindUsedBlobs(0xc15640, 0xc42020e9c0, 0xc1d760, 0xc42021fa40, 0x4edfb5f894134ac9, 0xf0bba160fd7f8f19, 0xe83800bc1f59b772, 0xc1b5cb4b8c32a833,
0xc44ee1c960, 0xc44ee1c990, ...)
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/internal/restic/find.go:31 +0x5a6 fp=0xc4d2cc7750 sp=0xc4d2cc74d8 pc=0x6fa366
github.com/restic/restic/internal/restic.FindUsedBlobs(0xc15640, 0xc42020e9c0, 0xc1d760, 0xc42021fa40, 0x5efbd8585a9898d2, 0x448917a1ace19706, 0x8ee237ec0622eb45, 0xc9d306cd39e8d05d,
0xc44ee1c960, 0xc44ee1c990, ...)
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/internal/restic/find.go:31 +0x5a6 fp=0xc4d2cc79c8 sp=0xc4d2cc7750 pc=0x6fa366
github.com/restic/restic/internal/restic.FindUsedBlobs(0xc15640, 0xc42020e9c0, 0xc1d760, 0xc42021fa40, 0x260fdd5acba993c3, 0x3ed04f4eeab05c9f, 0xe5ef8da60b35f98, 0xd090ad85e968ef5f,
0xc44ee1c960, 0xc44ee1c990, ...)
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/internal/restic/find.go:31 +0x5a6 fp=0xc4d2cc7c40 sp=0xc4d2cc79c8 pc=0x6fa366
github.com/restic/restic/internal/restic.FindUsedBlobs(0xc15640, 0xc42020e9c0, 0xc1d760, 0xc42021fa40, 0x60f7906c006bf615, 0x1bd2d33fc9768849, 0xee593a9983a1f5bf, 0x9ab809619e8cc152,
0xc44ee1c960, 0xc44ee1c990, ...)
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/internal/restic/find.go:31 +0x5a6 fp=0xc4d2cc7eb8 sp=0xc4d2cc7c40 pc=0x6fa366
github.com/restic/restic/internal/restic.FindUsedBlobs(0xc15640, 0xc42020e9c0, 0xc1d760, 0xc42021fa40, 0x8f00b9047475f470, 0x2094345d9e4dbb28, 0x3905971b8623f05c, 0x14b3bcdc097489ee,
0xc44ee1c960, 0xc44ee1c990, ...)
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/internal/restic/find.go:31 +0x5a6 fp=0xc4d2cc8130 sp=0xc4d2cc7eb8 pc=0x6fa366
main.pruneRepository(0x7ffc3e67bcea, 0x39, 0x7ffc3e67bd34, 0x58, 0x0, 0x0, 0x0, 0x7ffc3e67bd99, 0x11, 0x0, ...)
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/cmd/restic/cmd_prune.go:191 +0xea3 fp=0xc4d2cc9060 sp=0xc4d2cc8130 pc=0x9d23e3
main.runForget(0x0, 0x0, 0x7, 0x4, 0x6, 0x1, 0x0, 0x0, 0x0, 0x0, ...)
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/cmd/restic/cmd_forget.go:241 +0xece fp=0xc4d2cc9970 sp=0xc4d2cc9060 pc=0x9ca82e
main.glob..func10(0xf59720, 0xc420276160, 0x0, 0xb, 0x0, 0x0)
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/cmd/restic/cmd_forget.go:24 +0xb5 fp=0xc4d2cc9b60 sp=0xc4d2cc9970 pc=0x9e4935
github.com/spf13/cobra.(*Command).execute(0xf59720, 0xc420282180, 0xb, 0xc, 0xf59720, 0xc420282180)
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/spf13/cobra/command.go:762 +0x468 fp=0xc4d2cc9c50 sp=0xc4d2cc9b60 pc=0x6bae28
github.com/spf13/cobra.(*Command).ExecuteC(0xf59980, 0xffffffffffffffff, 0x0, 0x19)
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/spf13/cobra/command.go:852 +0x30a fp=0xc4d2cc9d90 sp=0xc4d2cc9c50 pc=0x6bb83a
github.com/spf13/cobra.(*Command).Execute(0xf59980, 0x23, 0xc4201cfeb8)
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/spf13/cobra/command.go:800 +0x2b fp=0xc4d2cc9dc0 sp=0xc4d2cc9d90 pc=0x6bb50b
main.main()
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/cmd/restic/main.go:86 +0x1d6 fp=0xc4d2cc9f88 sp=0xc4d2cc9dc0 pc=0x9e3ac6
runtime.main()
        /usr/lib/go-1.10/src/runtime/proc.go:198 +0x212 fp=0xc4d2cc9fe0 sp=0xc4d2cc9f88 pc=0x42ccc2
runtime.goexit()
        /usr/lib/go-1.10/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc4d2cc9fe8 sp=0xc4d2cc9fe0 pc=0x457e61

goroutine 5 [syscall, 92 minutes]:
os/signal.signal_recv(0x0)
        /usr/lib/go-1.10/src/runtime/sigqueue.go:139 +0xa6
os/signal.loop()
        /usr/lib/go-1.10/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
        /usr/lib/go-1.10/src/os/signal/signal_unix.go:28 +0x41

goroutine 6 [chan receive, 92 minutes]:
github.com/restic/restic/internal/restic.init.0.func1.1()
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/internal/restic/lock.go:257 +0xef
created by github.com/restic/restic/internal/restic.init.0.func1
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/internal/restic/lock.go:254 +0x35

goroutine 7 [select, 92 minutes, locked to thread]:
runtime.gopark(0xbb6a48, 0x0, 0xb84e9f, 0x6, 0x18, 0x1)
        /usr/lib/go-1.10/src/runtime/proc.go:291 +0x11a
runtime.selectgo(0xc42003ef50, 0xc420086240)
        /usr/lib/go-1.10/src/runtime/select.go:392 +0xe50
runtime.ensureSigM.func1()
        /usr/lib/go-1.10/src/runtime/signal_unix.go:549 +0x1f4
runtime.goexit()
        /usr/lib/go-1.10/src/runtime/asm_amd64.s:2361 +0x1

goroutine 8 [chan receive, 92 minutes]:
github.com/restic/restic/internal/restic.init.2.func1(0xc420086180)
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/internal/restic/progress_unix.go:17 +0xa3
created by github.com/restic/restic/internal/restic.init.2
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/internal/restic/progress_unix.go:16 +0xab

goroutine 10 [IO wait, 2 minutes]:
internal/poll.runtime_pollWait(0x7f7d7829fbc0, 0x72, 0xc42004abe0)
        /usr/lib/go-1.10/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc4202146a8, 0x72, 0xffffffffffffff01, 0xc0ef60, 0xf14738)
        /usr/lib/go-1.10/src/internal/poll/fd_poll_runtime.go:85 +0x9b
internal/poll.(*pollDesc).waitRead(0xc4202146a8, 0xc4b4f12001, 0x4, 0x4)
        /usr/lib/go-1.10/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc420214690, 0xc4b4f120c4, 0x4, 0x4, 0x0, 0x0, 0x0)
        /usr/lib/go-1.10/src/internal/poll/fd_unix.go:157 +0x17d
os.(*File).read(0xc42020a140, 0xc4b4f120c4, 0x4, 0x4, 0x18, 0x0, 0x0)
        /usr/lib/go-1.10/src/os/file_unix.go:226 +0x4e
os.(*File).Read(0xc42020a140, 0xc4b4f120c4, 0x4, 0x4, 0x18, 0x0, 0x0)
        /usr/lib/go-1.10/src/os/file.go:107 +0x6a
io.ReadAtLeast(0xc0d480, 0xc420144ae0, 0xc4b4f120c4, 0x4, 0x4, 0x4, 0xa6e780, 0xc42013a301, 0xc4b4f120c4)
        /usr/lib/go-1.10/src/io/io.go:309 +0x86
io.ReadFull(0xc0d480, 0xc420144ae0, 0xc4b4f120c4, 0x4, 0x4, 0xc42004aea8, 0x4056e7, 0xc42d7cc420)
        /usr/lib/go-1.10/src/io/io.go:327 +0x58
github.com/pkg/sftp.recvPacket(0xc0d480, 0xc420144ae0, 0x4051c5, 0xc42d7cc420, 0xc42004af70, 0x1, 0x985f32, 0xc445504001)
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/pkg/sftp/packet.go:143 +0x86
github.com/pkg/sftp.(*conn).recvPacket(0xc420144ae0, 0xc42004af70, 0xc4000378eb, 0xc445504108, 0x1, 0x0, 0x0)
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/pkg/sftp/conn.go:22 +0x37
github.com/pkg/sftp.(*clientConn).recv(0xc420144ae0, 0x0, 0x0)
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/pkg/sftp/conn.go:64 +0xe5
github.com/pkg/sftp.(*clientConn).loop(0xc420144ae0)
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/pkg/sftp/conn.go:49 +0x5b
created by github.com/pkg/sftp.NewClientPipe
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/pkg/sftp/client.go:128 +0x1c3

goroutine 18 [chan receive, 92 minutes]:
main.CleanupHandler(0xc4201fa060)
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/cmd/restic/cleanup.go:62 +0x5d
created by main.init.0
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/cmd/restic/cleanup.go:24 +0x6d

goroutine 19 [IO wait, 92 minutes]:
internal/poll.runtime_pollWait(0x7f7d7829ff00, 0x72, 0xc420038c70)
        /usr/lib/go-1.10/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc420214568, 0x72, 0xffffffffffffff01, 0xc0ef60, 0xf14738)
        /usr/lib/go-1.10/src/internal/poll/fd_poll_runtime.go:85 +0x9b
internal/poll.(*pollDesc).waitRead(0xc420214568, 0xc4202b8001, 0x1000, 0x1000)
        /usr/lib/go-1.10/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc420214550, 0xc4202b8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/lib/go-1.10/src/internal/poll/fd_unix.go:157 +0x17d
os.(*File).read(0xc42020a120, 0xc4202b8000, 0x1000, 0x1000, 0x0, 0x0, 0xc420038d78)
        /usr/lib/go-1.10/src/os/file_unix.go:226 +0x4e
os.(*File).Read(0xc42020a120, 0xc4202b8000, 0x1000, 0x1000, 0x1000, 0x1000, 0x0)
        /usr/lib/go-1.10/src/os/file.go:107 +0x6a
bufio.(*Scanner).Scan(0xc420038f30, 0xc115a0)
        /usr/lib/go-1.10/src/bufio/scan.go:208 +0xbf
github.com/restic/restic/internal/backend/sftp.startClient.func1(0xc115a0, 0xc42020a120, 0xb827c8, 0x3)
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/internal/backend/sftp/sftp.go:53 +0x1e6
created by github.com/restic/restic/internal/backend/sftp.startClient
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/internal/backend/sftp/sftp.go:51 +0x1b9

goroutine 20 [syscall, 92 minutes]:
syscall.Syscall6(0xf7, 0x1, 0x2035, 0xc420039598, 0x1000004, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/lib/go-1.10/src/syscall/asm_linux_amd64.s:44 +0x5
os.(*Process).blockUntilWaitable(0xc4202326c0, 0x0, 0x0, 0x0)
        /usr/lib/go-1.10/src/os/wait_waitid.go:31 +0x98
os.(*Process).wait(0xc4202326c0, 0x0, 0x0, 0x0)
        /usr/lib/go-1.10/src/os/exec_unix.go:22 +0x3c
os.(*Process).Wait(0xc4202326c0, 0x0, 0x0, 0x0)
        /usr/lib/go-1.10/src/os/exec.go:123 +0x2b
os/exec.(*Cmd).Wait(0xc420238580, 0x0, 0x0)
        /usr/lib/go-1.10/src/os/exec/exec.go:461 +0x5c
github.com/restic/restic/internal/backend/sftp.startClient.func2(0xc420238580, 0xc420144a80)
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/internal/backend/sftp/sftp.go:76 +0x2f
created by github.com/restic/restic/internal/backend/sftp.startClient
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/internal/backend/sftp/sftp.go:75 +0x29e

goroutine 34 [select, 2 minutes]:
main.refreshLocks(0xf65c00, 0xc420136060)
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/cmd/restic/lock.go:72 +0x221
created by main.lockRepository
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/cmd/restic/lock.go:49 +0x373

goroutine 98 [select, 15 minutes]:
github.com/restic/restic/internal/restic.(*Progress).reporter(0xc44ee282c0)
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/internal/restic/progress.go:169 +0x130
created by github.com/restic/restic/internal/restic.(*Progress).Start
        /build/restic-XDCNVl/restic-0.9.3+ds/obj-x86_64-linux-gnu/src/github.com/restic/restic/internal/restic/progress.go:96 +0x132

这不是一个永久性的解决方案,但是我做的是-由于restic如此“便携”-只是在公共云中启动VM(我使用了DigitalOcean,但是AWS或其他应该做的很好)。 给它一堆vCPU和RAM(我想我用了16个内核和64 GB的ram或其他任何东西),然后在那里进行了恢复。

它运行了几个小时,花了我仅几美元,但至少完成了。

清除旧内容后,我现在可以在常规服务器上运行“忘记--prune”。

这就要求您的备份服务器可以从公共云中获取。

这就要求您的备份服务器可以从公共云中获取。

不幸的是,不是。谢谢。

你好

即使服务器有足够的内存,我也遇到同样的问题。 我正在备份约150T,并尝试删除一些快照。 错误和内存利用率显示在随附的屏幕截图中。

运行Restic 0.9.3。
prune_outofmemory.txt

screenshot from 2019-01-20 11-24-52
screenshot from 2019-01-20 11-24-23

我正在运行这个:

restic 0.9.4 (1107eef2) compiled with go1.10.4 on linux/arm

在只有512MB RAM的ReadyNAS Duo v1上。 我总共有130190个文件@ 689.646 GiB(根据Restic输出)存储在B2中。

修剪需要一段时间(〜32小时),但可以顺利完成。
我所做的一件事是设置以下内容:

$ export GOGC=33
$ restic --option b2.connections=3 <other options here are omitted>

由于我的存储库位于云中,因此问题可能出在所有设置上吗?

这里发生了非常奇怪的事情……虽然我不能指出这一点。 在我看来prune正在使用过多的内存,但是我完全不知道发生这种情况的地方...

如果有人想花一些时间分析Restic并确定其存储库使用过多内存的原因,将不胜感激,这将有助于加快修复速度。

重新启动后,我仍然可以使用0.93进行修剪,尽管速度仍然很慢并且使用了一些内存

screenshot from 2019-01-22 11-59-29

screenshot from 2019-01-22 11-59-14

如果有人可以准确地告诉我他们需要我做些什么,我愿意做一些分析和测试。
在具有8 GB RAM的VM上,后端是Amazon S3,大约有500 GB数据。

@Olen感谢您提供帮助。 您需要具有Go方面的经验才能获得有用的结果,并且该过程是迭代的,相当复杂并且难以用文本描述。 不幸的是,尽管有很大的意图,但我们并没有为您作为用户提供很大帮助:)

有趣的是:在修剪过程的哪个阶段,内存使用失控了?

对我来说,发生了当restic checkprune.同时运行时

尽管到目前为止看起来prune仍在运行,没有内存问题。

那么restic check在每个prune操作之前运行?
screenshot from 2019-01-23 18-16-13

那么RESTIC检查是否在每次修剪操作之前运行?

我不确定我是否正确理解,您正在运行restic check然后是restic prune

这是两个不同的操作,尽管prunecheck做类似的事情,但是prune有几个阶段(构建索引,查找引用的数据,重写,删除文件,再次构建索引) ),找出在特定阶段内存使用量是否增加是很有趣的...

值得的是,我还在一些不同的服务器上看到了这一点,只是运行了一个检查:

restic check --no-lock
using temporary cache in /tmp/restic-check-cache-300914548
repository a63c6698 opened successfully, password is correct
created new cache in /tmp/restic-check-cache-300914548
load indexes
check all packs
check snapshots, trees and blobs
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0xc0d786, 0x16)
    /usr/local/go/src/runtime/panic.go:608 +0x72
runtime.sysMap(0xc0f0000000, 0x4000000, 0x11e41d8)
    /usr/local/go/src/runtime/mem_linux.go:156 +0xc7
runtime.(*mheap).sysAlloc(0x11ca7a0, 0x4000000, 0x11ca7b8, 0x7ff95adeb1e0)
    /usr/local/go/src/runtime/malloc.go:619 +0x1c7
runtime.(*mheap).grow(0x11ca7a0, 0x1, 0x0)
    /usr/local/go/src/runtime/mheap.go:920 +0x42
runtime.(*mheap).allocSpanLocked(0x11ca7a0, 0x1, 0x11e41e8, 0x400)

etc etc

这是在具有200GB存储库,4GB RAM和当前有2.3GB可用空间的服务器上。

我也遇到了内存问题,但是有check 。 我首先运行prune没问题(在130快照,200GB存储库中减少到17个快照和100GB),但随后的检查用尽了内存(尝试两次,结果相同):

using temporary cache in /data/backup/restic/restic-check-cache-567709739
created new cache in /data/backup/restic/restic-check-cache-567709739
create exclusive lock for repository
load indexes
check all packs
check snapshots, trees and blobs
fatal error: runtime: out of memory

然后,它为每个goroutine打印回溯跟踪(看起来可疑的数目很多,grep计数了其中的90个,尽管我猜每个实际的工作线程通常都关联着少数goroutine)。

这是在较旧的Atom系统上使用的(4GB RAM,大约2.5G可用空间,交换似乎已满)。 我正在从Debian测试中运行Restic 0.9.4 + ds-1版本。

不知道这是否实际上是与OP相同的问题,但它们似乎足够相关以添加一些信息。

这些实际上是有价值的数据点:这意味着我们可以更轻松地调试此问题(因为check不会修改存储库)。 谢谢!

当前主服务器中的常规内存使用率已经降低,并且一些其他优化正在进行中。

关于主要主题: @ johndoe31415您能否确认当前的master仍然存在此问题?

如果是,如果这样可以改善您的状况,您可以尝试#2718吗?
如果否,则应关闭IMO这个问题。

@aawsome-不幸的是我不能。 我最初用来重现此错误的机器已经过了一段时间的反调试,并迁移到了具有不同结构的存储库的另一台计算机,该存储库从未出现过所描述的问题。 对此我感到抱歉。

此页面是否有帮助?
4 / 5 - 1 等级

相关问题

ikarlo picture ikarlo  ·  4评论

fd0 picture fd0  ·  3评论

fd0 picture fd0  ·  4评论

christian-vent picture christian-vent  ·  3评论

RafaelAybar picture RafaelAybar  ·  3评论