Fabric: Issue with environmental variables & PATH on remote server (not sourcing .bashrc)

This is a bash thing, not a fab thing.

What files bash decides to source at startup can be complicated http://blog.flowblok.id.au/2013-02/shell-startup-scripts.html and debian/ubuntu (and most distros) have some customization in /etc/profile and in the default ~/.bashrc.

The default shell fab uses is /bin/bash -l -c, and the -l makes it a "login" shell. Without the debian/ubuntu customization, it's possible for a bash "login" shell to source ~/.bash_profile but not ~/.bashrc.

But on ubuntu 16.04, it does seem to source .bashrc by default even for login-shell. But lines added at the bottom of a default .bashrc are not processed, because it bails out near the top if it detects running non-interactively.

Here I've added two lines to the ubuntu-16.04 default user .bashrc

# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples

export VAR1=val1

# If not running interactively, don't do anything
case $- in
    *i*) ;;
      *) return;;
esac

export VAR2=val2

# don't put duplicate lines or lines starting with space in the history.
# See bash(1) for more options
HISTCONTROL=ignoreboth
...

Here's a fabfile I'm testing with:

from fabric.api import run, env, task

@task
def get_myvars():
    run("echo VAR1=$VAR1 VAR2=$VAR2")

The results:

$ fab -H testpy05.ec2.st-av.net get_myvars
[testpy05.ec2.st-av.net] Executing task 'get_myvars'
[testpy05.ec2.st-av.net] run: echo VAR1=$VAR1 VAR2=$VAR2
[testpy05.ec2.st-av.net] out: VAR1=val1 VAR2=
[testpy05.ec2.st-av.net] out: 
...

Sounds legit at a glance, thanks @ploxiln!

Thank you for the detailed answer!

This seems to differ from standard ssh behaviour and was hard to debug. When I manually ssh'd in all worked fine, where as fabric had a different PATH value due to not running .bash_profile in OSX.

@bitprophet this may be an OSX bug??

@code-tree Are you using Fabric 1 or 2? 1.x uses explicit shell wrappers which would be why it'd differ from standard OpenSSH client; version 2 doesn't do any wrapping and should behave a lot more like OpenSSH client in terms of what the sshd is running on your behalf.

Version 1 has some env config value options for changing that shell wrapper, so you may want to try adding flags to it, like -i which IIRC is bash for "run interactive mode and source some extra files".

That's strange because I'm using version 2.2.1. I currently have to do the following to get my app working:
c.run('bash -l -c "python3 ./configure.py"')
As when I installed Python 3 it added itself to PATH via my .bash_profile, which is not being run by Fabric. I haven't made any modifications to the standard builtin sshd config either.

That is strange then, I'll have to see if I can replicate. Just did a quick sanity trace to prove what I mean re: Fabric 2 not doing anything 'special' re: sshd driven execution:

• Runner.run is the start, takes command
• calls Runner._run_body, which is here, still taking it as an explicit kwarg
• _run_body then just calls Runner.start
• Fabric's Remote runner's .start is here
• which just calls Channel.exec_command with no additional modifications to command
• Paramiko takes over from there and just constructs a vanilla SSH exec channel request w/ the command inlined
• after which it should be completely identical to ssh myserver <command>, presumably

@code-tree also, if you can, please post some more details about exactly what you're running & other environmental details (client and server OS/version, etc etc).

Sure, I'm running an old OSX version as server. If you can confirm all works fine on a newer MacOS then oh well. But if not then maybe an issue with MacOS as server in general?

• Client: Ubuntu 16.04, Python 3.6.6, OpenSSH 7.2p2, Bash 4.3.48, Fabric 2.2.1
• Server: OSX 10.11, Python 3.6.6, OpenSSH 6.9p1, Bash 3.2.57

Side note, I'm wondering if this is actually a case of misdiagnosis:

fabric had a different PATH value due to not running .bash_profile in OSX

Could this instead be an example of #1744, where Fabric 2 is differing from ssh in terms of shunting local env vars down the pipe? (Depends on the exact env vars in question and whether they might be similar locally and remotely; @code-tree could easily disprove this theory by double checking the exact values in play re: their local and remote bash_profile... 😁)

But I'm going to troubleshoot this straight anyway, just so I know I'm not crazy re: my understanding of what the sshd is doing in terms of shells and sourced files.

Also I wonder if this is related to the scenario/intelligence from #1816, which is also about shells and startup files. It's _probably_ not directly relevant because of the assertion here about ssh behaving one way and Fabric behaving another way (versus it being "just how the sshd is invoking the shell normally"), but worth linking anyways.

Found that trying to stumble on a bunch of deep sshd code diving I remember doing in the last few months, related to something along these lines. I can't find it now though which is irritating. EDIT: ah I think it was an unrelated Paramiko issue (not linking because no point confusing things) about executing commands before authentication. OK.

All right, so what's really happening when we request command exec?

• SSH-the-protocol wants you to either make a shell channel request (which loads up the user's defined login shell program and does not take any command) or an exec channel request (which just says "executes given command string, which may contain a path" and doesn't specify more)
• Paramiko does the latter, as we found above.
• So what's OpenSSH itself actually doing? I'll be looking at my local checkout of openssh-portable (at 775f8a23f2353f5869003c57a213d14b28e0736e)

Well, first I'll goof around a little. Running against an arbitrary Debian 8.10 container (which is running OpenSSH 6.7!) I see this in the DEBUG3-level logs upon doing ssh localhost whoami:

debug1: server_input_channel_req: channel 0 request exec reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req exec
Starting session: command for root from 172.17.0.1 port 42598

Doing fab -H localhost -- whoami:

debug1: server_input_channel_req: channel 0 request exec reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req exec
Starting session: command for root from 172.17.0.1 port 42610

OK, they're both doing exec...that's to be expected. Anything different in the process tree? shouldn't be, and...there is not. both look like this:

sshd,1 -D -e
  └─sshd,1535
      └─pstree,1537 -alpU

So, definitely no shell in play? What's it even doing? Can I use shelly things like &&?

I can! eg whoami && id works fine. So that's a bit weird given my understanding of Unix process control; it seems unlikely the sshd is just literally doing exec(string), but if it were using a shell to interpret, wouldn't we see that in pstree?

I think it's time to really dig into openssh-portable and see what's what.

• All the magic seems to happen in session.c
• First interesting stop is where the session_input_channel_req log line happens, in (you guessed it) session_input_channel_req, which logs and then dispatches to various functions depending on session type (shell, pty open, exec, subsystem, etc)
• In our case it's exec which means session_exec_req
• Which just punts us over to do_exec
• Which in turns just logs again and eventually kicks us to either do_exec_pty or (in our case, because we did not request a pty; you can tell because the log lines don't have a bit about the tty number) do_exec_no_pty
• Which forks and then calls do_child
• Which does in fact get a shell and set up its environment!
• and does things with rc files...
  
  
  
  • except it just means SSH's own .rc files
  
  
• finally, yup, it uses that shell it set up to run execve with the shell, -c, and the command string

Also, huh, I _did_ do part of this tracing (but for a different reason/focus) in the past, as in, Dec 2016: https://github.com/paramiko/paramiko/pull/398#issuecomment-264281759. Had totally forgotten...

The reason why I'm not seeing the shell in pstree is because of the use of execve, which does a replacement of parent process instead of spawning a child. Good to know. (EDIT: wrong, that just replaces the sshd child proc with the shell; the shell itself may then also be doing a replace-me style exec call; see below comments.)

Anyway, so! This still should mean Fabric and OpenSSH's clients are doing the same exact thing in terms of shell execution; nothing either of them is doing would be modifying these parts of OpenSSH's codebase. It should always be something like bash -c "python3 ./configure.py", with the actual value of bash depending on @code-tree's macOS level user and their configured shell.

Makes me wonder if my hunch about env var transmission being the differentiator is accurate, since that's the only other thing I can think of re: how the two execution environments could differ. FWIW in my CLI-focused testing, env reports same on both systems, and regardless of those other env related issues, the _default_ behavior (due to SSH security policy) should always be that no local env "leaks" to the other side.

Yeah, I'm pretty sure sshd always runs the command string in the user's shell (configured in /etc/passwd). However it prepends the command string with "exec" if it can parse the command string and determine that it's a single command.

$ ssh testdeploy02.ec2.st-av.net pstree -a
...
  |-sshd -D
  |   `-sshd 
  |       `-sshd  
  |           `-pstree -a

$ ssh testdeploy02.ec2.st-av.net 'pstree -a && sleep 1'
...
  |-sshd -D
  |   `-sshd 
  |       `-sshd  
  |           `-bash -c pstree -a && sleep 1
  |               `-pstree -a

$ ssh testdeploy02.ec2.st-av.net 'VAR=-a sh -c "exec pstree \$VAR"'
...
  |-sshd -D
  |   `-sshd 
  |       `-sshd  
  |           `-pstree -a

As for environment variables, these are mostly controlled by the ssh client and sshd server configs:

$ grep Env /etc/ssh/*_config
/etc/ssh/ssh_config:    SendEnv LANG LC_*
/etc/ssh/sshd_config:AcceptEnv LANG LC_*

So the ssh client does explicitly send some variables (outside of the command-string itself) and they are filtered by sshd. But there are apparently some exceptions:

$ ssh testdeploy02.ec2.st-av.net env | grep TERM
$ ssh -t testdeploy02.ec2.st-av.net env | grep TERM
TERM=xterm-256color

Actually I think the "automatic exec to replace shell with single command" behavior is a feature of bash:

$ dash -c "pstree -a"
...
  ├─sshd -D
  │   └─sshd 
  │       └─sshd  
  │           └─bash
  │               └─dash -c pstree -a
  │                   └─pstree -a

$ bash -c "pstree -a"
  ├─sshd -D
  │   └─sshd 
  │       └─sshd  
  │           └─bash
  │               └─pstree -a

EDIT: For completeness:

$ bash -c "pstree -a && sleep 1"
  ├─sshd -D
  │   └─sshd 
  │       └─sshd  
  │           └─bash
  │               └─bash -c pstree -a && sleep 1
  │                   └─pstree -a

Ah yea, I was mistaken, execve just means the sshd child proc is what gets replaced, so what happens after that is up to the shell itself and what it does for -c xxx. My tests were in zsh, fwiw.

Also @ploxiln yea the thing with env vars is referencing other tickets linked above, though it remains to be seen if @code-tree is truly experiencing a different symptom to those.

Sorry, I think this is a false alarm. I wasn't aware that there was any difference between executing a command via ssh inline and executing after logging in via ssh. When I do ssh host 'echo $PATH' it also has not updated PATH where as echo $PATH after logging in works fine.

I still don't really get why ssh does this (seems like I have to indeed do ssh host 'bash -l -c "python3 ./configure.py"'). Regardless, it is safe to say this isn't a Fabric issue after all. Sorry for the confusion.

@code-tree it's related to what was mentioned earlier - shells have a few different modes they operate in, frequently called 'login' and 'interactive' (and usually those are on top of a base mode which is considered neither) and which mode the shell is being run in, changes which set of rc-files it loads.

The salient point here is that bash -c xxx isn't considered "interactive" and thus skips loading certain files - like .bash_profile - but just running bash without -c is usually interactive, and does load profile files.

As you saw above - sshd always runs bash -c <command send down the pipe> when you ask it to run a single command (as Fabric does, or as ssh host command does), and thus, is always in non-interactive mode. (Unless you do what Fabric 1 does and run your own, nested, shell with -l...but then you're running bash -c "bash -l -c \"oh god escaping is hard help\"" and I'm going to go have some 🥃 now.

Anyway, not all is lost, I need to refresh my memory of this part of things every few years apparently so uh...now I'm refreshed 😆

This is a bash thing, not a fab thing.

What files bash decides to source at startup can be complicated http://blog.flowblok.id.au/2013-02/shell-startup-scripts.html and debian/ubuntu (and most distros) have some customization in /etc/profile and in the default ~/.bashrc.

The default shell fab uses is /bin/bash -l -c, and the -l makes it a "login" shell. Without the debian/ubuntu customization, it's possible for a bash "login" shell to source ~/.bash_profile but not ~/.bashrc.

But on ubuntu 16.04, it does seem to source .bashrc by default even for login-shell. But lines added at the bottom of a default .bashrc are not processed, because it bails out near the top if it detects running non-interactively.

Here I've added two lines to the ubuntu-16.04 default user .bashrc

# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples

export VAR1=val1

# If not running interactively, don't do anything
case $- in
    *i*) ;;
      *) return;;
esac

export VAR2=val2

# don't put duplicate lines or lines starting with space in the history.
# See bash(1) for more options
HISTCONTROL=ignoreboth
...

Here's a fabfile I'm testing with:

from fabric.api import run, env, task

@task
def get_myvars():
    run("echo VAR1=$VAR1 VAR2=$VAR2")

The results:

$ fab -H testpy05.ec2.st-av.net get_myvars
[testpy05.ec2.st-av.net] Executing task 'get_myvars'
[testpy05.ec2.st-av.net] run: echo VAR1=$VAR1 VAR2=$VAR2
[testpy05.ec2.st-av.net] out: VAR1=val1 VAR2=
[testpy05.ec2.st-av.net] out: 
...

You saved my life. This is perfect answer I am finding:) Many thanks!