Restic: Rclone making most of the backends redundant

This doesn't even make any sense. Rclone is a transfer utility, not a backup utility. It cannot replace restic repositories.

He probably just means ditching restic's own backends code, except the one for rclone, so that the rclone integration becomes the one that all backends go through.

@jtagcat I think you mean "backends" instead of "repositories".

I agree with this proposal in spirit.
Actually removing all other backends may be too disruptive, but at least documentations can highlight rclone as the preferred backend, and label other remote backends as "legacy".

Actually I think we can do better, then marking backends as legacy.
In docs, mark them as legacy, but in the inner works, translate legacy to rclone on the fly. On some rclone backends, you can directly translate (an example from the top of my head is http), others you translate to a temporary config file, then use --config /path/to/config with rclone.

If you ask me as a user of restic, I'd say having to use rclone for all backends would be horrible. It requires setting up rclone configs for your backends and keeping an rclone binary, the former which very much removes the ad-hoc and config-free nature of restic that is really nice. The one thing I don't like about rclone is that I have to configure it before running it, instead of just being able to give it two URIs as the source and target.

A better suggestion IMO is if we could embed rclone in restic, such that you can keep using restic the same was as you do now, but with the backend support that rclone provides. If that was to happen, it'd be fine to ditch current backend code assuming the corresponding rclone code provides the same features.

How would you embed rclone in restic? If this issue was to be positively closed, it would have to be done.
Would you be like 'rclone not found, running curl https://rclone.org/install.sh | sudo bash' (curl and sudo might not exist on system)? Because you can't simply add it as a dependency in package managers, if you are not installing via a pkg manager.

Translating current backends to temporary rclone configs on the fly is possible, when using purely rclone for stuff, I'm a bit annoyed by that myself as well.

How would you embed rclone in restic?

I haven't investigated what's possible in that regard. It's just something I've thought about a few times.

Thought about it a bit more, embedding would probably end up as 'installing as a dependency', to allow rclone to be up-to-date without restic needing to be updated.
Probably the best way would be to add rclone as a dependency, where possible and have 'rclone not found, installing' everywhere (when rclone dependency is not found for some reason, as a backup).

Though the rclone installer presents a few problems. Make it posix-compatible.

• curl - I have seen a system or two, where it wasn't installed out of the box.
• sudo - same as above
• unzip/other decompressing thing, you probably have experienced this, while installing rclone on a minimal installation.

I'm thinking more in terms of making rclone embeddable as al library or whatever so it can be integrated in restic. Shouldn't be a problem to release a new restic version when there's something in rclone that is updated to the extent that it warrants a new release.

Probably the best way would be to add rclone as a dependency, where possible and have 'rclone not found, installing' everywhere (when rclone dependency is not found for some reason, as a backup).

Sorry to say that, but trying to (automatically) install software on the user's machine is (probably) the worst idea that I ever read in restic's issue discussions.

Because:

• The environment may not allow installation of new software.
• The user (or the system administrator) may not want new software to be installed.
• Accessing the Internet without explicit request may be undesirable.
• The installer may guess a wrong place for the new software.
• Installation of new software may interfere with the normal functions of the environment (for example, by providing executables outside of regular places where users/package managers/administrators/auditors/whatever else expect to find them, and thus creating confusion).
• The installation may fail or produce incorrect results; either in an obvious manner, or in a subtle and hard-to-discover way.

Probably the best way would be to add rclone as a dependency, where possible and have 'rclone not found, installing' everywhere (when rclone dependency is not found for some reason, as a backup).

Sorry to say that, but trying to (automatically) install software on the user's machine is (probably) the worst idea that I ever read in restic's issue discussions.

Because:

* The environment may not allow installation of new software.

* The user (or the system administrator) may not want new software to be installed.

* Accessing the Internet without explicit request may be undesirable.

* The installer may guess a wrong place for the new software.

* Installation of new software may interfere with the normal functions of the environment (for example, by providing executables outside of regular places where users/package managers/administrators/auditors/whatever else expect to find them, and thus creating confusion).

* The installation may fail or produce incorrect results; either in an obvious manner, or in a subtle and hard-to-discover way.

yes, what do you suggest?

what I meant was like this:

restic rclone not found...
Install 'rclone' with command 'curl https://rclone.org/install.sh | sudo bash'  to provide rclone backends? [Y/n]

I would not replace existing backends with rclone, unless the latter is successfully embedded as a library, and thus always included in restic's distribution.

Until this is done (or instead of doing this), I'd make rclone an optional dependency in package managers that allow this (for example, apt and portage). If rclone isn't installed, restic could show a message referring to rclone's Web page, suggesting to either (1) consult the page for ways to install, (2) install rclone using system's package manager, (3) issue the "curl + bash" command, with a warning that "make sure you know what you are doing".

I would not execute the command on user's behalf, even after asking about it.

The nature of RESTIC (especially on linux) as a static binary with no dependencies has been ABSOLUTELY INVALUABLE in bare-metal restore and recovery scenarios.

I currently prefer to use the "rest-server" (with haproxy frontend for authentication) for in-house backups and depend on restic supporting it natively. If I was forced to mess with RESTIC and RCLONE during a time critical restore, I'd probably find another solution.

The nature of RESTIC (especially on linux) as a static binary with no dependencies has been ABSOLUTELY INVALUABLE in bare-metal restore and recovery scenarios.

I currently prefer to use the "rest-server" (with haproxy frontend for authentication) for in-house backups and depend on restic supporting it natively. If I was _forced_ to mess with RESTIC _and_ RCLONE during a time critical restore, I'd probably find another solution.

Rest backend wouldn't be removed, as it's unique to Restic. I as well use rest-server, it's very convenient to just plop the credentials in.

I think the issue is in the state of 'only, if rclone is embedded'. As nobody really needs this (though embedded rclone would be neet). I guess right now, it's more 'don't add new backends, what aren't already supported by rclone', and a feature request for somebody (probably in many, many months, me), who bothers to do this, embed rclone.

As for bare-bones, need to get alternative releases out, that don't use bzip (as far as I've heard, the only reason of installing bzip is restic). As for that, if nobody else does, I probably will, since a todo for me is #2705

The one thing I don't like about rclone is that I _have_ to configure it before running it, instead of just being able to give it two URIs as the source and target.

@rawtaz
You are able to do just that since rclone does not require you to configure anything.
You can completely remove any _~/.rclone.conf_ or _~/.config/rclone/rclone.conf_, they are just consulted for defaults or fallback options.
Now try this:

rclone copy ./file.txt :sftp:subdir --sftp-host=localhost --sftp-key-file=~/.ssh/id_rsa

It's perfectly valid and described in the docs: https://rclone.org/docs/#backend-path-to-dir

You can also all pass options via environment or mix it with CLI arguments:

export RCLONE_SFTP_HOST=localhost
export RCLONE_SFTP_KEY_FILE=~/.ssh/id_rsa
rclone ls :sftp:

Using rclone as a library is discussed in https://github.com/rclone/rclone/issues/633