Restic: MacOS Mojave: error: Open: ... operation not permitted

Yeah, apparently you have to do the Full Disk Access thing: https://www.backblaze.com/blog/mojave-permissions/

Are you _sure_ you added the right restic binary? Did you move the file or change any of its attributes (ownership, etc) after adding it to Full Disk Access in System Preferences?

(I have a Mac, but I don't yet have Mojave, sorry.)

I use homebrew, so first I added /usr/local/bin/restic to Full Disk Access, started a job, noted the same errors, so then I removed that and added the path to the actual binary (noting that this would unfortunately have to be redone every time restic updates): /usr/local/Cellar/restic/0.9.2/bin/restic, unfortunately I see the same errors.

No changes, added binary to Full Disk Access, switched back to terminal and ran sudo myscript.sh, which worked for most files but gave permission errors on others.

Side-note, there is another (possibly Mojave-related) issue I'm trying to flesh out, where my sftp pubkey authentication stopped working when run from launchctl (as root), but works when run manually as root, but I'll file a new issue if I'm able to determine that it's not specific to my setup.

Interesting. What happens if you run restic directly? And/or add your script to FDA. Just shooting in the dark here, honestly. If I had Mojave I'd try it out.

Good thought.

My script itself is not executable (for no good reason really), as noted I run it with sudo bash myscript.sh (/usr/local/bin/bash actually); it can't be added to FDA because it is not executable.

I tried adding /usr/local/bin/bash to FDA and no dice.

EDIT: I'm wrong apparently, even after chmod +x, my backup.sh still can't be added directly to FDA (while the bash and restic binaries can). Odd.

EDIT2: Just to be thorough, I added the bash and restic binaries both to FDA and I see the same error.

Probably a bigger issue than restic -- I even tried adding /bin/ls to the FDA list and still get an error.

$ sudo /bin/ls /Users/me/Library/Suggestions
ls: Suggestions: Operation not permitted

EDIT: Ugly workaround: add Terminal.app to the FDA permissions and the permissions errors go away.

Relevant thread: https://forums.developer.apple.com/thread/107546

Tried codesigning both the restic binary and my backup.sh script and adding to FDA, no luck.

(After reading some of that thread) My goodness. This is super annoying. Thanks for looking into it! I'll keep investigating too once I upgrade...

Wow, thank you very much for the information and all the time you spent on the problem! I don't have a mac at all, so I'm glad if you both could figure it out and we can document this issue for other users! Thank you again!

@mholt If you want to, you can install a 30 day trial of VMware Fusion and install Mojave in that (unless Apple crippled the ability to install it ad-hoc). That trial doesn't do any pollution and can be uninstalled if you don't want it later on.

I noted above that adding Terminal.app to the System Preferences -> Security & Privacy -> Full Disk Access list was a workaround, since when I manually ran my script (sudo /usr/local/bin/bash mybackup.sh) it seemed to work without the permissions errors.

For some reason, when I checked my logs this morning from the automated overnight restic run (based on a launchd script at /Library/LaunchDaemons/com.n8henrie.restic.plist which runs nightly with root privileges), the permissions errors are still there.

And unfortunately, I am now unable to get the workaround to work -- even with Terminal.app in FDA, I still get the permissions errors when I run sudo launchctl start com.n8henrie.restic or sudo /usr/local/bin/bash mybackup.sh.

So the workaround doesn't seem to work, or something has changed.

EDIT: Gah, just added all 3 to FDA -- Terminal.app, /usr/local/bin/bash, and /usr/local/Cellar/restic/0.9.2/bin/restic -- and now it ran without errors. 🤷‍♂️

FWIW, here is my log output, which contains the list of files getting errors. As you can see, there are some big concerns, like my photos library.

@n8henrie From the Mac support thread you linked to earlier, it seems that FDA requires the approved apps to be a registered .app bundle in the Applications folder, which might be why Terminal.app succeeds... but you have to add all 3 to FDA in your case? Interesting...

@mholt it looks like something else is going on. I left my settings exactly as they were yesterday (all 3 in FDA, which produced no permissions errors), and my overnight run still got the same old permissions errors.

This has happened twice now, where after tinkering for a while I get runs with no permissions errors that then reappear. Does restic somehow know whether or not a file has changed without needing to open the file -- some kind of cache? I wonder if I'm getting fooled by runs that are done closely together, and there have been no interim changes to a file and therefore restic doesn't try to open it?

Otherwise I'm having a hard time explaining what's going on here.

Good question. I know restic uses a cache but I have not looked into how aggressively it adheres to it in the case of permissions errors, etc.

Relevant discussion at SO: https://apple.stackexchange.com/q/338213/27415

Good question. I know restic uses a cache but I have not looked into how aggressively it adheres to it in the case of permissions errors, etc.

Not at all. The cache only contains files from the repo, no information about the file system (that is not in the repo) is included.

Okay -- yeah, I don't think this is a bug in restic. This is definitely a macOS Mojave issue that, at this point, I'm pretty sure restic itself can't do anything to fix.

Cool, thanks for the feedback, I'm closing this issue for now. Please add more comments if you find out things. Thanks!

I'm a little surprised to see the issue closed already -- this seems like a
major incompatibility with a major platform, and closing it at this point
may put it "out of sight, out of mind."

I understand that this is not primarily a restic issue, but it seems like
there are some reasonable steps that could be taken for MacOS users. As is,
the ability of a user to back up much of their most important personal
data (e.g. photos) is compromised by default on the current version of
MacOS. That seems like a big deal to me, for any backup software!

Some suggestions / possibilities (that I'm happy to help work on):

• Possibilities for fixing the problem
  
  
  
  • Is it possible to provide a proper MacOS wrapper application which
    
    can get added to the FDA list, which contains the restic binary
  
  
  • If able to figure out how to do the above, instead of providing the
    
    application, could we include instructions for how users can accomplish it
    
    themselves with some sort of Makefile or XCode?
  
  
• Workarounds
  
  
  
  • Provide info in the docs regarding the most secure / minimal binaries
    
    that need to be added to FDA
  
  
  • Provide links to info regarding (and risks of) disabling SIP
  
  
• Warnings
  
  
  
  • Include info in the docs somewhere that Mojave users will not have
    
    their data fully backed up by default
  
  

Overall, the situation seems not terribly dissimilar to full backup without root
on Linux, with disabling SIP altogether being something like running as
root, and determining and recommending a reasonably secure compromise with
FDA being something like using setcap.

Unfortunately my Macbook is in the shop this week so I won't be able to
work on any of these right away, but I'd be interested to hear thoughts
from others; perhaps I'm way off base, or possibly ignorant of specifics of
the restic team workflow for managing issues.

A few more updates now that I have my machine back:

I cannot replicate my successes above in getting a full system backup from my launchd script.

I've tried adding all of the below to the FDA list (at the same time), and still get the errors:

• restic
• bash
• launchctl
• launchd
• Terminal.app

Just for experimentation, bare binaries seem to work fine when added to the FDA list and seems to have nothing to do with codesigning. Compare the ls binaries provided by MacOS and by Homebrew:

$ codesign -d /bin/ls
Executable=/bin/ls
$ codesign -d /usr/local/opt/coreutils/libexec/gnubin/ls
/usr/local/opt/coreutils/libexec/gnubin/ls: code object is not signed at all

Before and adding to the FDA list:

$ /bin/ls ~/Library/Mail
ls: Mail: Operation not permitted
$ /usr/local/opt/coreutils/libexec/gnubin/ls ~/Library/Mail
ls: cannot open directory '/Users/n8henrie/Library/Mail': Operation not permitted
$ # Added to FDA
$ /bin/ls ~/Library/Mail
PersistenceInfo.plist V6
$ /usr/local/opt/coreutils/libexec/gnubin/ls ~/Library/Mail
PersistenceInfo.plist  V6

Additionally, they work fine when put into a bash script as long as ls is in FDA (no need to add bash separately), which makes me think I should only need to add restic.

Also, for comparison running the following Go script errors if not in FDA, but works fine by itself as well as when called from launchd as long as the binary is in FDA (no need to add launchd / launchctl / anything else).

package main

import (
    "fmt"
    "io/ioutil"
)

func main() {
    matches, err := ioutil.ReadDir("/Users/n8henrie/Library/Mail")
    if err != nil {
        fmt.Println("Err:", err)
    } else {
        for _, match := range matches {
            fmt.Println(match.Name())
        }
    }
}

Output:

$ ./gotest
Err: open /Users/n8henrie/Library/Mail: operation not permitted
$ # Add to FDA
$ ./gotest
.DS_Store
PersistenceInfo.plist
V6

Next I'll do some experimenting with Restic to see if I can figure out why it's getting permission errors even when added to FDA (although other binaries seem to be working once added).

Ok, thanks for the feedback! I'll reopen this issue, maybe we can figure it out what's going on and then add some documentation to the manual.

~Still getting Open errors on a fresh repo, with restic added to the FDA list.~

See edit below.

$ restic -r /tmp/restic backup ~/Library/Mail -vvv
open repository
enter password for repository:
repository 9ccb5357 opened successfully, password is correct
created new cache in /Users/me/Library/Caches/restic
lock repository
load index files
start scan on [/Users/me/Library/Mail]
start backup on [/Users/me/Library/Mail]
scan: Open: open /Users/me/Library/Mail: operation not permitted
scan finished in 1.849s: 0 files, 0 B
can not obtain extended attribute com.apple.quarantine for /Users/me/Library/Mail:
error: Open: open /Users/me/Library/Mail: operation not permitted
new       /Users/me/Library/, saved in 0.012s (0 B added, 13 B metadata)
new       /Users/me/, saved in 0.012s (0 B added, 381 B metadata)
new       /Users/, saved in 0.013s (0 B added, 379 B metadata)

Files:           0 new,     0 changed,     0 unmodified
Dirs:            3 new,     0 changed,     0 unmodified
Data Blobs:      0 new
Tree Blobs:      4 new
Added to the repo: 1.119 KiB

processed 0 files, 0 B in 0:01
snapshot 4a658c73 saved
$ restic -r /tmp/restic ls latest
enter password for repository:
repository 9ccb5357 opened successfully, password is correct
snapshot 4a658c73 of [/Users/me/Library/Mail] filtered by [] at 2018-11-04 11:30:05.334024 -0700 MST):
/Users
/Users/me
/Users/me/Library

EDIT: Disregard this comment -- I'm still plagued by intermittent errors that are confounding any progress, perhaps in association with updating to MacOS 10.14.1 yesterday. Today, the same Go code from yesterday that consistently worked with FDA and failed without it (I toggled it on and off multiple times to be sure) is no longer working even with FDA. Same with ls.

It does work if Terminal.app is added to FDA (as the sole entry), which wasn't necessary yesterday.

🤷‍♂️

I'll report back if I can find anything on this in the Apple forums.

EDIT2: Wife's Macbook is still on 10.14, and /bin/ls ~/Library/Mail does not work with /bin/ls added to FDA, so seems like it may not be something new in 10.14.1. 😕

Still working on this.

It continues to be pretty painful, but I finally have a workaround that I think will work for my purposes.

I've detailed the process here, but the bottom line is:

You can use Script Editor.app to make an AppleScript into an application bundle. That AppleScript can run your restic backup script (in my case /path/to/restic-backup.sh, where restic-backup.sh is a bash script that runs restic with my desired settings), and you can add the resulting application bundle to FDA in order to get access to the protected files.

However, this makes it difficult to automate running backups on a system level. The built-in open command works, but runs the application under your user -- so while the FDA protected files above are backed up, you'll get permissions errors on any files that are only root readable (e.g. root-owned 0600 stuff).

My workaround at this point is to have the AppleScript call the backup script with sudo and give my user privileges to run that script with root privileges without a password (sudo visudo, NOPASSWD:, etc.).

It's not pretty, but it seems to be working.

I'd like to leave this open for any suggestions from MacOS users with more expertise. If nothing more satisfying comes up, I could add this info to the docs (although this solution honestly doesn't seem very satisfactory).

Unbelievable, this seems to work and is far cleaner and more simple.

// Runrestic provides a binary to run my restic backup script in MacOS Mojave with Full Disk Access
package main

import (
    "log"
    "os"
    "os/exec"
    "path/filepath"
)

func main() {
    ex, err := os.Executable()
    if err != nil {
        log.Fatal(err)
    }
    dir := filepath.Dir(ex)
    script := filepath.Join(dir, "restic-backup.sh")
    cmd := exec.Command("/usr/local/bin/bash", script)
    if err := cmd.Run(); err != nil {
        log.Fatal(err)
    }
}

The resulting binary can be chown root and chmod 0700, then add to Full Disk Access, and it seems to work. It can then be added to a /Library/LaunchDaemons plist to run automatically.

First 2 runs are working so far, I'm hopeful that this doesn't end up like several false starts above.

• https://apple.stackexchange.com/a/343260/27415

My automated run last night worked. I'm deploying this strategy to my wife's Macbook Air today, if it looks like it works there as well, I'll consider this a reasonable fix and work on a small PR to https://github.com/restic/restic.net (if that seems reasonable).

Hi @n8henrie, I came here after hitting this exact problem and then finding your blog post. Thanks for doing all this research.

The above solution (call shell script from simple Go binary) isn't working for me. Are you sure it's successfully accessing all your files?

I notice in particular that stdout/stderr is discarded to /dev/null. It also cannot read stdin if, for example, restic wants to prompt for a password. (Also kind of funny, why is your bash /usr/local/bin/bash and not /bin/bash? Just curious.)

Anyway I made the following changes to see the error output:

    cmd := exec.Command("/bin/bash", script)
    cmd.Stdin = os.Stdin
    cmd.Stdout = os.Stdout
    cmd.Stderr = os.Stderr

Before seeing your blog post my first instinct was to add the restic binary itself to the FDA, and that did not work for me under 10.14.1 (18B75). I'm not sure why inserting another program (the Go wrapper calling a shell script, ultimately calling restic) would change anything.

Is this still working for you?

@n8henrie thanks for keeping us posted! You could also write a blog post about it on the restic blog if you're up for it (in addition to a small section in the manual or so)... :)

@fd0 I'd be honored!

@armhold yes, seems to be working like a charm, see below. On my wife's MBA, I think I required a reboot before it started working (not on mine though). IIRC for hers, I created the binary, then rebooted, then added to FDA, and it worked.

Before the fix:

Thu Nov 15 02:00:00 MST 2018 :: Starting restic-backup.sh
can not obtain extended attribute com.apple.rootless for /Library/Application Support/com.apple.TCC:
error: Open: open /Library/Application Support/com.apple.TCC: operation not permitted
error: open /Library/Preferences/com.apple.TimeMachine.plist: operation not permitted
error: Open: open /Users/me/Library/Application Support/AddressBook: operation not permitted
error: Open: open /Users/me/Library/Application Support/CallHistoryDB: operation not permitted
error: Open: open /Users/me/Library/Application Support/CallHistoryTransactions: operation not permitted
error: Open: open /Users/me/Library/Application Support/MobileSync: operation not permitted
error: Open: open /Users/me/Library/Application Support/com.apple.TCC: operation not permitted
error: Open: open /Users/me/Library/Calendars: operation not permitted
error: Open: open /Users/me/Library/Containers/com.apple.Home: operation not permitted
error: Open: open /Users/me/Library/Containers/com.apple.Safari: operation not permitted
error: Open: open /Users/me/Library/Containers/com.apple.VoiceMemos: operation not permitted
error: Open: open /Users/me/Library/Containers/com.apple.iChat: operation not permitted
error: Open: open /Users/me/Library/Containers/com.apple.mail: operation not permitted
error: Open: open /Users/me/Library/Containers/com.apple.news: operation not permitted
error: Open: open /Users/me/Library/Containers/com.apple.stocks: operation not permitted
can not obtain extended attribute com.apple.quarantine for /Users/me/Library/Cookies:
error: Open: open /Users/me/Library/Cookies: operation not permitted
error: Open: open /Users/me/Library/HomeKit: operation not permitted
error: Open: open /Users/me/Library/IdentityServices: operation not permitted
can not obtain extended attribute com.apple.quarantine for /Users/me/Library/Mail:
error: Open: open /Users/me/Library/Mail: operation not permitted
error: Open: open /Users/me/Library/Messages: operation not permitted
error: Open: open /Users/me/Library/Metadata/CoreSpotlight: operation not permitted
error: Open: open /Users/me/Library/Metadata/com.apple.IntelligentSuggestions: operation not permitted
can not obtain extended attribute com.apple.metadata:com_apple_backup_excludeItem for /Users/me/Library/PersonalizationPortrait:
error: Open: open /Users/me/Library/PersonalizationPortrait: operation not permitted
error: open /Users/me/Library/Preferences/com.apple.AddressBook.plist: operation not permitted
error: open /Users/me/Library/Preferences/com.apple.AddressBook.plist.KaSTvBv: operation not permitted
error: open /Users/me/Library/Preferences/com.apple.AddressBook.plist.M410OmB: operation not permitted
error: open /Users/me/Library/Preferences/com.apple.AddressBook.plist.Sjhd5Xh: operation not permitted
error: open /Users/me/Library/Preferences/com.apple.AddressBook.plist.ceAM0im: operation not permitted
error: open /Users/me/Library/Preferences/com.apple.homed.notbackedup.plist: operation not permitted
error: open /Users/me/Library/Preferences/com.apple.homed.plist: operation not permitted
error: open /Users/me/Library/Preferences/com.apple.mail-shared.plist: operation not permitted
error: Open: open /Users/me/Library/Safari: operation not permitted
can not obtain extended attribute com.apple.metadata:com_apple_backup_excludeItem for /Users/me/Library/Suggestions:
error: Open: open /Users/me/Library/Suggestions: operation not permitted
can not obtain extended attribute com.apple.FinderInfo for /Users/me/Pictures/Photos Library.photoslibrary:
can not obtain extended attribute com.apple.quarantine for /Users/me/Pictures/Photos Library.photoslibrary:
error: Open: open /Users/me/Pictures/Photos Library.photoslibrary: operation not permitted

Files:         179 new,   261 changed, 857338 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 266.392 MiB

processed 857778 files, 186.192 GiB in 12:57
snapshot 46831f24 saved
Thu Nov 15 02:12:58 MST 2018 :: restic-backup.sh finished.
Duration: 778 seconds

After the fix:

Tue Nov 27 02:00:00 MST 2018 :: Starting restic-backup.sh

Files:         389 new,  2367 changed, 1055845 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Added to the repo: 430.279 MiB

processed 1058601 files, 295.471 GiB in 18:16
snapshot e58d8f1c saved
Tue Nov 27 02:18:17 MST 2018 :: restic-backup.sh finished.
Duration: 1097 seconds

Same fix is also working on my wife's Macbook Air, both verified on the remote with restic find '/Users/*/Library/Mail' --snapshot latest --host=$(hostname) (where ~/Library/Mail is one of the directories usually protected, as can be seen in the log of errors above).

I notice in particular that stdout/stderr is discarded to /dev/null. It also cannot read stdin if, for example, restic wants to prompt for a password.

This will depend entirely on your script. As you can see in my original post, since my setup is fully automated, it is set up to read password from a (root-owned 0600) file, but could also read from an envvar or any of the usual methods. You're right, I don't think this will work for interactive stuff. It also logs everything to a file: &>> /path/to/file.log.

(Also kind of funny, why is your bash /usr/local/bin/bash and not /bin/bash? Just curious.)

I use Homebrew to get a more recent version of bash

$ /bin/bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin18)
Copyright (C) 2007 Free Software Foundation, Inc.
$ /usr/local/bin/bash --version
GNU bash, version 4.4.23(1)-release (x86_64-apple-darwin17.5.0)
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

This gets me more modern features, such as the &>file.log operator (which saves a few keystrokes compared to 2>&1 >file.log.

Before seeing your blog post my first instinct was to add the restic binary itself to the FDA, and that did not work for me under 10.14.1 (18B75). I'm not sure why inserting another program (the Go wrapper calling a shell script, ultimately calling restic) would change anything.

I'm also not entirely sure about this. For my use case, I have a lot of other setup that I do in my bash script (somewhat complicated sftp command where the destination, the backup paths, and several options are based on the hostname) so calling the restic binary itself wasn't an option. I can experiment with it a bit.

OK, thanks for the explanation. I just tried rebuilding, rebooting + adding FDA, and I still get operation not permitted. Also tried moving both the binary wrapper and the shell script into /Applications, and no luck. I'll keep digging.

Huh. Definitely doesn’t need to be in /Applications for me.

Are you on 10.14 or 10.14.1? I’m on the latter.

And you might try using the tccutil command I referenced to clear out all entries beforehand.

I'm on 10.14.1 (18B75). I tried the tccutil command too, no luck. I understand why Apple is doing this, but it really is frustrating to not have a clear path about what to do here.

Huh. This is odd.

Oddly, I tried and I can't get the restic binary to work directly.

Unclear why this is not working but my workaround is (for me).

Seems like another intermittent error -- I think we should hold off on
formal documentation / writeup until we're able to get someone else's
computer working with the same system. I'm 2 for 2, but I don't see why
this isn't working for @armhold.

@armhold, can you post a copy of the exact bash script and Go code you're
using and how you're running it? I'll see if I can reproduce on my end.

Oddly, I tried and I can't get the restic binary to work directly.

Yeah, that's what I don't understand. I don't see why the Go wrapper would succeed where the restic binary itself would fail. Something seems off.

can you post a copy of the exact bash script and Go code you're using and how you're running it?

Sure, here it is: https://github.com/armhold/restic-fda.

• both the shell script and the go wrapper are installed in my ~/bin directory.
• added ~/bin/restic-fda to FDA via System Preferences
• I'm running restic-fda from a non-root user account interactively on the command line in Terminal. I get errors like: error: Open: open /Users/armhold/Library/Application Support/AddressBook: operation not permitted.
• running as root (via sudo) fails similarly

I have not tried running it via launchctl.

Little Snitch, a firewall, does something similar - for outgoing connections, it would ask whether to allow "Terminal via restic" do make the connection (rather than just restic); i.e. Terminal.app being the primary one to grant permission to.

What I did in the end was to bundle my shell script as an app using Platypus, and granting FDA to that generated app. Then launch backups via Finder. Works so far.

launchctl will be the next step.

FYI this is still working for me (including after most recent MacOS
update). Any news from @armhold?

Still does not work for me. I gave up and just gave Terminal FDA.

Unfortunately, I'm stumped again.

I'm on 10.14.2. My restic backup script is still working, no permissions errors in its logs. However, I now can't get @armhold's script or even my prior test scripts to work (that do nothing but open the protected directory ~/Library/Mail).

🤷‍♂️

Have you tried running your (working) wrapper on a fresh repo? What I mean is, are you sure it's not skipping files that are old, and therefore already present in the repo? I imagine you'd still get an error trying to descend into the protected folders though, as restic walks the tree.

@mholt What's the status on this and relica - have you worked around it somehow, and if so can you share with us what steps you take to do so?

After upgrading a couple of clients I'm seeing this behavior too, very annoying and hard to deal with without wrapping stuff and messing around.

@rawtaz

What's the status on this and relica - have you worked around it somehow, and if so can you share with us what steps you take to do so?

I only upgraded to Mojave just last week myself. But I installed Relica and was able to reproduce the "operation not permitted" error when trying to backup ~/Library/Mail.

I then added Relica.app to the FDA screen and re-ran the Relica backup.

It backed up successfully without errors this time (whereas the first snapshot made by Relica+restic did not show the Mail folder at all because of the permission error):

So I am afraid I don't have any answers to contribute to this thread. :-/ I'm not sure if it'll _keep_ working but of course I hope it does.

I too settled on wrapping my restic backup script in a .app bundle to be able to give it FDA. It sucks having to do this, but it seems to be the only practical way forward.

At first I tried giving just the restic binary FDA, but that did not make it work.
I also tried giving my backup script (a Bash script that calls restic) FDA, but that didn't work either.
I did not try @armhold's wrapper.

I used Platypus as suggested above, and it works great. It produces an .app bundle that I can then give FDA in the system settings of macOS, and this resolves the issue.

The only downside I've noticed during use is that when the .app starts (for me it's started by crontab using open -ga ~/Applications/Backup.app, the current window loses focus. This can be a serious annoyance for my users, but it is what it is. At least we have working backups again. I thought the -g switch would take care of that, but it doesn't change anything unfortunately.

I very briefly tried to see what happens when you delete the .app bundle (move it to Trash, empty trash) and replace it with a new .app bundle having the same name. My observations are that when you remove the old one, the entry in FDA in system preferences disappears, but when you put the new one back in the same place, the entry appears again, indicating that the system recognizes and would consider the new .app to have FDA. However, when I ran the new app after that, I got the original errors back. Once I removed the FDA entry for the app in system preferences and added a new FDA entry for it, the errors were gone again. So for now, I am presuming that when replacing the .app bundle, I need to replace the FDA entry as well, for it to work. It might very well be that if one just replace parts of the .app bundle, it will keep working though. This needs further investigaton AFAICT.

For anyone who wants to bundle restic into a .app bundle, here's a somewhat generic tutorial I wrote a few months ago about how to do it for any Go program, including the code if you just want to change a few settings and be on your way: https://medium.com/@mattholt/packaging-a-go-application-for-macos-f7084b00f6b5

10.14.3 -- still no good news for running with binary only.

$ /bin/ls ~/Library/Mail/
ls: : Operation not permitted

Works if Terminal.app is added to FDA, but not with just ls (or other binaries).

applescript/automator works but shows an icon in dock; alternatively, using xcode/swift cli, compile this into a binary and add that to the FDA (replace /full/path/to with your real paths)

import Foundation
import os

let task = Process()

task.launchPath = "/full/path/to/bash"
task.arguments = ["/full/path/to/backup_script.sh"]

do{
    try task.run()
}
catch{
    os_log("error")
}

task.waitUntilExit()

@daviehh no luck here.

import Foundation
import os

let task = Process()

task.launchPath = "/bin/ls"
task.arguments = ["/Users/me/Library/Mail"]

do{
    try task.run()
}
catch{
    os_log("error")
}

task.waitUntilExit()

$ swiftc foo.swift
$ ./foo
ls: Mail: Operation not permitted
$ # add to FDA
$ ./foo
ls: Mail: Operation not permitted
$ sudo ./foo
ls: Mail: Operation not permitted

I recently started using Restic and have been trying to get it to work as a cron job called from the root crontab sudo crontab -e so that I could feel a little more secure having my backup script in a file only accessible with sudo privileges. I was getting the exact same errors as @n8henrie but now I have a working solution and would like to know if this works for others here.

First a little background of my setup:

I have a backup script in /Users/myuser/bin named restic-backup.sh with permissions 700 (only root/sudo read/write/execute). I execute this file with my root crontab sudo crontab -e. I use iTerm as my default terminal. I have installed restic and zsh with Homebrew.

macOS version 10.14.5

restic-backup.sh:

My backup script file has the following in it.

#!/usr/local/bin/zsh
restic_path="/usr/local/bin"
logFile="/Users/myuser/Documents/Backups/configurations/Mac/backup_logs/$(date +%F_%H%M)_restic.log"
unset HISTFILE
export RESTIC_REPOSITORY="..."
export AWS_ACCESS_KEY_I'd="..."
export AWS_SECRET_ACCESS_KEY="..."
export RESTIC_PASSWORD="..."
$restic_path/restic --verbose backup /Users/myuser  &> $logFile

Then in my root crontab file: sudo crontab -e I have:

0 */2 * * * /Users/myuser/bin/restic-backup.sh

In Full Disk Access:

cron ==> /usr/sbin/cron
iTerm.app ==> /Applications/iTerm.app

Like @n8henrie, I thought that the programs actually accessing the files like restic would be the ones that require FDA but instead it seems that the programs making the initial sudo request need the FDA: cron in the automated case, and iTerm.app in the manual case.

I discovered something useful: when whitelist an app, it applies to any script or binary inside the app's directory. So, you don't have to launch the app directly. In fact, the app itself is completely irrelevant -- it simply acts as a container for whitelisting. You can copy your script to an arbitrary app, whitelist that app, then run your script. The only caveat is that you have to remove and re-add the app to the whitelist every time you change your script or binary within it.

@atticusmatticus @russelldavis -- I think one of the recent MacOS updates may have changed something again -- I had definitely tried both of those strategies with no luck. But I had also tried the below, which had stopped working, but is now working again (kind of):

package main

import (
    "fmt"
    "io/ioutil"
)

func main() {
    fmt.Println("Starting...")
    matches, err := ioutil.ReadDir("/Users/me/Library/Mail")
    if err != nil {
        fmt.Println("Err:", err)
    } else {
        for _, match := range matches {
            fmt.Println(match.Name())
        }
    }
}

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
        <key>Label</key>
        <string>com.me.gotest</string>
        <key>ProgramArguments</key>
        <array>
            <string>/Users/me/go/src/github.com/me/gotest/gotest</string>
        </array>
        <key>StartInterval</key>
        <integer>15</integer>
        <key>StandardErrorPath</key>
        <string>/Users/me/go/src/github.com/me/gotest/stderr.txt</string>
        <key>StandardOutPath</key>
        <string>/Users/me/go/src/github.com/me/gotest/stdout.txt</string>
    </dict>
</plist>

1. go build
2. Add the gotest binary (and nothing else) to Full Disk Access
3. copy the plist to ~/Library/LaunchAgents/
4. Load the launchd daemon: launchctl load -w ~/Library/LaunchAgents/com.me.gotest.plist

Now, curiously, with the gotest binary, but not launchd (or launchctl) added to FDA, I still can't run directly:

$ ls -l gotest
-rwxr-xr-x 1 me staff 2142552 Jul  2 09:15 gotest
$ ./gotest
Starting...
Err: open /Users/me/Library/Mail: operation not permitted
$ sudo ./gotest
Password:
Starting...
Err: open /Users/me/Library/Mail: operation not permitted

But it is running without errors from the launchd daemon (every 15 seconds as configured), which is not running as root (~/Library/ not /Library/, and launchctl load not sudo launchctl load):

$ cat stdout.txt | head
Starting...
.DS_Store
PersistenceInfo.plist
V6
Starting...
.DS_Store
PersistenceInfo.plist
V6
Starting...
.DS_Store

Curiously, I am still seeing, including a reboot:

||go binary in FDA|go binary not in FDA
---|:---:|:---:
go binary without sudo|err|err
go binary with sudo|err|err
launchd running go binary|RUNS|err

Not directly related restic, but to the FDA problem.
I think I've got an idea how it is probably working and I hope I'm not been captain obvious. I yet failed to find helpful discussion, article or documentation about.

FDA works for applications that were whitelisted (opening means that they are launched by launchd), for any children processes of those applications even not whitelisted ones and for whitelisted binaries if the are directly launched by launchd (launchd.plist or launchctl submit). It doesn't work if whitelisted binary is launched in a process tree with non whitelisted parent application/binary.

From this it looks like it is launchd that is responsible for whitelisting process tree depending on application/binary been whitelisted.

From experiments it looks like the path to binary should be exact, so whitelisting will not be triggered if for whitelisted binary launchctl plist has WorkingDirectory specified and relative path ./some-binary is used, it will not even work for /some/path/./some-binary or /some/path/../path/some-binary, only /some/path/some-binary.
It is also not possible to use whitelisted application for shebang even if launching script directly by launchd, so #!/some/path/some-binary will not work, only /some/path/some-binary /path/to/script.

Yes, I run from launchd. Some interesting reads here

https://eclecticlight.co/2018/10/02/how-privacy-protection-is-enforced-through-the-attribution-chain/

https://eclecticlight.co/2018/09/06/working-with-mojaves-privacy-protection/

I think we have enough information and data points to provide at least some guidelines (and perhaps a link to this issue) in the docs, and I'd like to start working on it if that's okay.

Should this be a new point under Examples, like the section on backing up without running as root on Linux?

I plan to note that:

• Full Disk Access is required to back up as much of the disk as possible
• A launchd script running as root can call a binary that is whitelisted under FDA without needing to add launchd to FDA
• Unless Terminal.app is listed in FDA, the binary cannot access the full disk when called from Terminal
• If Terminal.app is listed in FDA, it can access the full disk when it calls a binary whether or not that binary is in FDA

Does this sound like it reflects everyone's understanding and experience?

While somewhat convoluted, I've been pleased with my setup running on 2 Macbooks over the last >1y, and I would probably include parts of it as an example. My setup is:

• I have a shell script that runs a machine-specific restic command based on various environment variables (the same script also runs on 4 Linux computers, which call this script directly).
• On my MacOS computers, I build a go binary that basically just runs the shell script above. I use a shell script here for easy configuration / updates / version control, though this could also be easily done in Go directly.
• This binary is added to FDA
• I run a system-wide launchd daemon (that runs with root permissions) to launch this go binary on a schedule

Using a binary similar to https://github.com/restic/restic/issues/2051#issuecomment-442872479 is working for me. I went with c as I don't have go installed right now. For others to copy/paste:

1. backup.c

#include <stdlib.h>
int main(void) {
  int status = system("./backup.sh");
  int ret = WEXITSTATUS(status);
  return ret;
}

1. compile: gcc -Wall -o backup backup.c
2. whitelist the backup binary and use it as you like

Curiously, I am still seeing, including a reboot:

go binary in FDA go binary not in FDA
go binary without sudo err err
go binary with sudo err err
launchd running go binary RUNS err

Thanks!

The solution for me was to create .plist file that calls restic directly and put all the parameters either inside or into separate files, using -p, --exclude-files, --files-from options. And of course give restic binary FDA permissions:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>my.backup_agent</string>

    <key>ProgramArguments</key>
    <array>
        <string>/usr/local/bin/restic</string>
        <string>backup</string>

        <string>-r</string>
        <string>s3:https://MY.STORAGE.SERVER/....</string>

        <string>-p</string>
        <string>.config/backup/restic.pwd</string>

        <string>--files-from</string>
        <string>.config/backup/backup.lst</string>

        <string>--exclude-file</string>
        <string>.config/backup/exclude.lst</string>
    </array>

    <key>EnvironmentVariables</key>
    <dict>
        <key>AWS_ACCESS_KEY_ID</key>
        <string>XXX</string>

        <key>AWS_SECRET_ACCESS_KEY</key>
        <string>YYY</string>
    </dict>

    <key>WorkingDirectory</key>
    <string>/Users/ME</string>

    <key>StandardErrorPath</key>
    <string>/Users/ME/log/backup.log</string>

    <key>StandardOutPath</key>
    <string>/Users/ME/log/backup.log</string>

    <key>StartCalendarInterval</key>
    <dict>
        <key>Hour</key>
        <integer>13</integer>

        <key>Weekday</key>
        <array>
        <integer>1</integer>
        <integer>2</integer>
        <integer>3</integer>
        <integer>4</integer>
        <integer>5</integer>
        </array>
    </dict>
</dict>
</plist>

How to find which app should be add to FDA? In short, find the app runs it.

You can keep the process running, and walk the parent pid of the process until ancestor pid is 1, launchd via ps ajx or ps ao pid,ppid,command with grep.

And in short:

• run via crontab, /usr/sbin/cron, deprecated by launchd.plist
• run via launchd.plist, the binary in Program or ProgramArguments
• run in Terminal, Terminal.app
• run via ssh, /usr/libexec/sshd-keygen-wrapper or similar
• run via other app, find it.

So, for @n8henrie , you need to find the actual binary.

Another way to address this if you're running things via launchd: LaunchControl now ships with a helper called fdautil that you can whitelist, and then run commands using fdautil exec. It will only allow commands you've whitelisted via LaunchControl or fdautil set.

There's a bit of info about it at https://www.soma-zone.com/LaunchControl/FAQ.html, and more details in the app's help window.