Restic: Implement web server to view backups

Created on 18 Nov 2014  ·  42Comments  ·  Source: restic/restic

Implement a restic command that starts a web server to browse snapshots.

user interface feature suggestion

Most helpful comment

I'd recommend a whole other application called something along the lines of restic-web written in go or nodejs and react or angular to be able to do everything you can do from the commandline interface

All 42 comments

This seems like a lot of complexity for something that doesn't seem to be the core responsibility of a backup program. Is this really needed? What's the use case? I would prefer FUSE (https://github.com/restic/restic/issues/27).

Windows doesn't have fuse. Having a basic web server is very little work using http://golang.org/pkg/net/http/#FileServer. This should take the same type of interfaces used to implement a fuse interface anyway.

Exactly what I thought, it's not much code and probably the only way to browse snapshots in Windows.

I agree. At some point it's time for a GUI, and I think the most natural step to that is an embedded web server which can present it on localhost or even other interfaces if wanted. Everyone has a web browser and one wouldn't have to fiddle around with applications.

@rakoo are you interested in working on the web server next?

Thoughts about the user interface:

  • Start the HTTP server with restic server
  • restic writes an HTTP URL to stdout which includes randomly generated username and password, e.g. http://RarOjHogvueHov:egsevtekwucvith@localhost/snapshots
  • The webserver requires HTTP digest auth
  • And it only binds to localhost

Binding to something else than localhost shouldn't be impossible, but really hard. Maybe we'll hardcode localhost for the first version, and see if users complain.

Thoughts?

This is more or less what I had in mind, except without the username/password: IMO they provide little value against a potential attacker (we'd do plain HTTP, so the username and password are clearly visible).

I'm fine with binding to localhost only: seeing the space we operate in, I expect user to have ssh to access their VPS, so I think it's fair to expect them to tunnel access to their repo this way:

backupserver> restic server 5000

clientmachine> ssh -Nf -L 5000:backupserver:5000
clientmachine> open http://localhost:5000/

I realise this is quite old, but +1 from me. I also don't think the embedded username/password is helpful as for me, this is all behind a VPN anyway and I would like it started at system-up time.

@yatesco What's your use case for starting up the discussed web server component at system startup time? I can't think of one, could you elaborate?

Hi @fd0 - it is to expose the repository to to the team.

Rather unusually maybe, I have a bunch of data which I don't want hanging around so using this archive as 'long term storage' is great, but then I need to provide access to it from the team and a samba share (e.g. #377) or a web ui would be great

Ok, thanks for the explanation.

Is it possible to expose the Fuse mount via Samba? I can't get it working, folder isn't visible. How do I get the Fuse mount to use "allow_other"?

I've outlined what needs to be done here: https://github.com/restic/restic/pull/754#issuecomment-276023059

Also see issue #767 for the fuse mount options.

Is it just a viewer we need ? or a full GUI for all tasks ? Here features I imagine for a GUI:

Restore features:

  • list snapshots (with date, snap size, real size, etc.)
  • browse a snapshot (sort, search, etc.)
  • view file version history
  • restore a snapshot, or a selection of folder/files to user local drive

Admin features:

  • remove snapshots
  • backup
  • check
  • rebuild

With Electron we keep the crossplatform/HTML5 GUI but with additional features like filesystem access, exec commands, open ssh connections, native modules, etc.

I can't speak for the project, but for me, the restore features would be sufficient.

I mostly work on Windows boxes SSH'ing into the servers and if things fail I'd like to quickly plug in my backup drive to view / download the files in question.

Hi gents, just started to work with restic, really good job, thx! My fifty cents on this topic. Just 'view snapshot' is a bit useless imho (can be done with fuse mount published by any means like nginx, samba, etc), definitely there should be some useful actions a user can do next, e.g. restore files to selected hosts, restore sql dump to some db host, restore anything that restic backed up in easy way. There are two main things: 1. Data search/classification and 2. Do something under the data (e.g. restore, delete, share, classify, compare copies(diff), etc.). Why not FUSE: it's not about Windows at all, it's about UX. Having a web frontend for backup store with rich search and easy restore (tailored to particular data type and system) is really cool. I imagine dozen use cases that make sense for backup store (advanced search, remote restore, backup policy suggestions, data masking, PII protection, shredding, data compliances, reporting, etc), having a web front for some of them (at least search and restore) would be nice! For backup where in most cases the main purpose is just to restore recent version of something, in case of data corruption, as fast as possible, it is good enough, but for functionality like archiving having a web frontend is a must. Btw data archive is another cool feature, e.g. detach certain data from restic snap and move it to long-term archive, but this is another story and off topic for the thread) And yea, if you decide to do this in future, imho this should be done in rest-server, wrapping core features in rest api. The latter makes sense for integration with 3rd party systems.

I'd recommend a whole other application called something along the lines of restic-web written in go or nodejs and react or angular to be able to do everything you can do from the commandline interface

Could restic utilize something like https://github.com/dokan-dev/dokany/ on windows?
I occasionally use cppcryptfs to access folders under Windows that were encrypted using gocryptfs under Linux and it seems to work reasonably well.

Hello,
I have found this article: https://devblogs.microsoft.com/commandline/announcing-wsl-2/
It says:

Now that WSL 2 includes its own Linux kernel it has full system call compatibility. This introduces a whole new set of apps that you can run inside of WSL. Some exciting examples are the Linux version of Docker, as well as FUSE!

Please, take into account that WSL requires Hyper-V to work, but:

1) Hyper-V is only available for some versions of Windows. For example, Windows 10 Pro has Hyper-V support, while Windows 10 Home does not.

2) Hyper-V cannot coexist with any other hypervisor software the user may already be using (like VirtualBox, etc). So, if someone is already using VirtualBox (or any another hypervisor), they will not be able to use WSL.

In summary, I think that WSL should not be taken for granted for every Windows user.

@aliron19 Can you cite a source for that? I am using WSL on my work pc to run redis, but this is a Windows 10 Home pc, so no Hyper-V and yet WSL works just fine.

see https://docs.microsoft.com/en-us/windows/wsl/wsl2-faq#does-wsl-2-use-hyper-v-will-it-be-available-on-windows-10-home

Thanks for clarifying. Seems that the Hyper V requirement is new to WSL 2, I thought aliron was talking about WSL in general. Also good to know Hyper V will become available for Windows 10 Home users.

I would say that we should prefer a Webdav server (#485) instead of a HTML interface.
This would also allow the users to easily add it as a network drive to the windows explorer..

If you got a server (or unix/linux) with nodejs, you can easily do that:

restic mount backup &
npx http-serve backup

Then you can browse http://127.0.0.1:8080
and see you files:

Capture d’écran 2020-07-24 à 17 58 18

When you are done you can:

CTRL + C # stop npx
fg # bring back restic
CTRL + C # stop restic

I don't know if restic locks the repository while you mount it, or if you can add more backup in the meantime.
But at least you can have a tiny web interface to browse your files.

@vincent-ogury I thought the point of this proposed feature is to provide a user friendly backup view where restic mount _isn't_ available.

You are right. This should work inside a docker on windows I guess.
A web version may be nice for other platforms.
It may simplified command line to show snapshots, view, forget, start a new backups...

I would like to work on this issue. The ideas is to implement a vanilla Go http server with some basic html template. Our use case is to have some universal basic interface for Restic under Windows
Here is the scope of my task:

Problem
Restic does not provide a handy way to browse through the backups on Windows.

Task
Try to implement a basic proof of concept of a built-in web server, that allows browsing and restoring the backups using a browser.

Example
Executing the following commands will create two backups and you learn how to browse the backups via the command line.

openssl rand -hex 10 > ~/.restic-password
restic -p ~/.restic-password init --repo /tmp/backup

Make a backup

restic -p ~/.restic-password --repo /tmp/backup backup ~

Make some changes

date > ~/file-with-changes.txt

Make the next backup

restic -p ~/.restic-password --repo /tmp/backup backup ~

Look what backups you have

restic -p ~/.restic-password --repo /tmp/backup snapshots

repository 2ad83051 opened successfully, password is correct

ID Time Host Tags Paths

b8ee7b24 2020-09-11 17:20:17 thorsten-devvm-v3 /root

99388d0b 2020-09-11 17:21:23 thorsten-devvm-v3 /root

2 snapshots

Look what what's inside a backup

restic -p ~/.restic-password --repo /tmp/backup ls -l 99388d0b

Get a json formatted list of all files in the backup.

restic -p ~/.restic-password --repo /tmp/backup ls -l --json 99388d0b|jq

Check if a file is in the backup

restic -p ~/.restic-password --repo /tmp/backup find snmp.json

Restore a file

restic -p ~/.restic-password --repo /tmp/backup restore latest --target /tmp/restore --include "/root/snmp.json"
Specifications
By executing something like that, the webserver starts, and internally the backup is "open".
restic -p ~/.restic-password --repo /tmp/backup webserver 127.0.0.1:3344

Pointing my browser to HTTP://127.0.0.1:3344 I get the list of all snapshots in the repository. It's the same data, that restic -p ~/.restic-password --repo /tmp/backup snapshots shows.

I can click on the ID on the first column and I get a tree-view of all files in the snapshot. The same data as restic -p ~/.restic-password --repo /tmp/backup ls -l --json returns is displayed. The tree-view doesn't need folding and unfolding nor icons at the beginning.

Or I can click on a path of a snapshot (5th column) and I get a list of files in the backup belonging only to this path. It displays the same data as restic -p ~/.restic-password --repo /tmp/backup ls -l --path --json

I can click on a single file and download it through the browser.

Ideally, I can click on a file and I get a list of all different versions ordered by date of that file.
A snaphot always gives a virtual view of the entire file system. To identify the different versions of a file the following command would be used on the command line.
restic -p ~/.restic-password --repo /tmp/backup find /root/snmp.json --json|jq
All objects, where mtime is identical must be reduced to a single object because files haven't changed.

IMHO adding support of "restic mount" on windows with something like dokany would make more sense to have the same restore workflow on Linux and Windows, and provide a better user experience using OS Explorer to browse the files.

Please see https://github.com/restic/restic/pull/2862 which addresses which is intended to solve Restic does not provide a handy way to browse through the backups on Windows in the same way as it's done on other platforms than Windows.

Sure but I guess it's quite hard for a "typical" windows user to use command line interface, my target is to provide a simple and understandable interface which any user can understand.

As a workaround, I've put together what @vincent-ogury and @celogeek suggested (thanks!) and made this dockerfile; https://github.com/stevedenman/restic-browser-docker
I've been running it on my Windows machine (in Docker on wsl2) and it works nicely for browsing snapshots and downloading files.

sorry guys do you say I should not do it? why there is a problem with that? the suggested solutions are quite hard for an average user in my case all we need to do - start program (with the defined flags) and open browser!

Hello.

As a fellow restic user, I just wanted to drop my two cents on the subject.

I'm a tech-savvy user so I'm fine with the way things are in restic. I'm also aware that restic is towards tech-savvy people; hence, the current solutions are/could be complex for some. But I see no problem with implementing a local web server that's accessible via the browser. It shouldn't replace the current solutions but as a side-solution it would be nice to have something like that. And perhaps, that would make restic more accessible to even more people.

Thanks.

I totally disagree with the "it's too hard for Windows users" opinions. First of all, these users have already managed to run restic to back their stuff up, and running the mount command instead of the backup command is no harder. Second, we're talking about opening the command line and entering restic -r foo mount bar here. Seriously, how is this hard for anyone? If someone is having problem with that, those people need to educate themselves. This is not something that is "hard", it's just one command line away.

EDIT: Just to clarify, I mean that this shouldn't be hard at all for any Windows user given restic mount working on Windows like in tended in #2862 - I'm not suggesting what @filippobottega wrote about right below this comment is easy for an average Windows user. I don't think one should have to use WSL to use restic mount.

Hello,
I have successfully mounted a restic backup on Windows Explorer using WSL2 but is not so easy as I'll expected.
You have to:

  1. Install WSL2 as explained in Windows Subsystem for Linux Installation Guide for Windows 10
  2. Change the default user of Ubuntu to root as explained in gurnec response at Can not edit /etc/nginx/nginx.conf from Windows. Access is denied.
  3. Logon to an Ubuntu session as root and mount the restic backup as explained in Restore using mount, for example restic -r /mnt/c/ResticBackup -p /mnt/c/ResticBackup/password-file.txt mount /mnt/restic

    At the end you will be able to see:

image

Second, we're talking about opening the command line and entering restic -r foo mount bar here. Seriously, how is this hard for anyone?

If you consider BIOS options to let hypervisor work, setting up WSL and diagnosing any problems coming from it malfunctioning, etc this can be quite laborious. In my view a requirement to run Microsoft Hypervisor that is incompatible with VmWare and VirtualBox I'm using, is plain ridiculous for restic.

Web browser point and click is much easier and also http is somewhat better supported (hint: browsers) than windows flavour of smb if you want to share in LAN. Also It's not clear if you can expose these mounts via smb at all, but even if you can, the point still stands.

In my view there is no question that the web browser option is simpler.

If you consider BIOS options to let hypervisor work, setting up WSL and diagnosing any problems coming from it malfunctioning, etc

@AndrewSav That's not what's being discussed here. See comment from @rawtaz (emphasis mine)

EDIT: Just to clarify, I mean that this shouldn't be hard at all for any Windows user _given restic mount working on Windows like in tended in #2862_ - I'm not suggesting what @filippobottega wrote about right below this comment is easy for an average Windows user. I don't think one should have to use WSL to use restic mount.

@cfbao you need to install a kernel driver for that. I'm always for an option that does not require additional kernel dirvers. I think this should be no question that running a program and opening your browser is simpler then installing some third party software, probably rebooting after that and then running some obscure command line (by that I mean you need to supply and type command line parameters, in browser you simply can click for navigation).

Installing third party drivers also has a clear disadvantage of impacting the whole system if there is a bug in said driver. Again, a requirement from restic for third party driver is not reasonable, if there is much simpler (browser) options.

As far as I can see these options are not contradicting, so restic can support both and thus cater for the needs of both groups.

@cfbao you need to install a kernel driver for that. I'm always for an option that does not require additional kernel dirvers.

@AndrewSav Point being that it's a very simple task that every Windows restic user would be able to do easily. The marginal utility of implementing a web server is hence greatly reduced and may not justify the cost to restic development and maintenance.

@AndrewSav The web interface is nice for looking around the backup, I can imagine that. But if there is a task to restore multiple files from different directories I'd prefer the decent file manager and backup snapshot mounted as a file system. If restic is to provide WebDAV interface to the back that is still inferior solution on Windows because of its WebDAV client limitations: 1) default max file size limit is 50M, user would need to do registry manipulations to extend it to 4G; 2) some characters are not supported in file names.
My opinion is that web interface and proper filesystem mount are complementary features with somewhat different use cases.
The third-party kernel driver (e.g. http://www.secfs.net/winfsp/) is need to be installed only once and there is no reboot required.
I also don't quite understand argument about "running some obscure command line". Wouldn't you need to run some restic command to activate web interface as well?

Thanks everyone for your input! So let's sum things up:

  • @fd0 has previously suggested and thereby approved implementing a HTTP serving repository browser in this issue and elaborated on it in https://github.com/restic/restic/issues/60#issuecomment-122898872 .
  • Lot's of people think it's a good idea to have this platform-neutral feature to browse snapshots and their files.
  • A common opinion is that we also want to be able to restore files from it, to the extent it's practical (we might not expect to be able to restore an entire archive through the web browser, it'd be more of a "downloading files" action).
  • It should not do any write operations on the repository, only read operations.
  • WebDAV, while being an option for browsing files, is problematic in terms of client support. We might implement WebDAV too at some point, but neither that nor a web server excludes the other one - they're separate features.

I don't see anything preventing continued work and a PR for this. Personally I support it. I think we should:

  • Start small, make a web GUI that can browse snapshots and files in it, then once we have a solid base start adding restore functionality.
  • Make it start using the command restic serve http so that we can later add other methods of serving this such as restic serve webdav and perhaps restic serve rest.
  • Make it generate a temporary username and password when starting the server, to
  • allow for configuring the IP/hostname and port to bind to using an argument, e.g. restic serve http 127.0.0.1:8080.
  • Not write ugly code that hardcodes assets/HTML/JS/CSS but instead embed a proper web GUI using e.g. https://github.com/tv42/becky or even start by just using net/http/#FileServer.
  • Realize that this is a feature that will take quite some time to finish, it's not something we'll do over night.

Existing work and similar references:

Was this page helpful?
0 / 5 - 0 ratings