I like the idea and find it useful, but think it would be far more useful (I'm thinking specifically in the case of a DSL) if I can get a single string expansion. So if I've built a cmd.exe compatible string in my PS, I have a way to string verbatim to cmd.exe. Perhaps

&n -command $string

That would potentially also let me specify my interpreter:

&n -shell /bin/sh -command $string
or
&n -shell cmd.exe -command $string

in terms of the name/operator, is "&!" in use? That has similarity to shbang( "#!" ).

$! is a good suggestion!

The shell can be specified by simply specifying it as the command:

&! /bin/sh -c blah

The $var expansion creates a problem in that $ may need to be literal to be passed to the native command/shell. One of the problems this tries to avoid is when people need to figure out how to properly escape everything. If you require variable expansion, you could always do:

$mycommand = "/bin/sh ls"
Invoke-Expression "&! $mycommand"

However, in cases where you want to mix PowerShell with the native command, it's probably best to use the current syntax and just be aware of what needs to be escaped properly as I would consider that advanced usage.

$mycommand = "/bin/sh ls"
Invoke-Expression "&! $mycommand"

I can't believe you just suggested iex. :-)

@essentialexch there are a few times where it's appropriate :)

Note that in the example, expectation is that the user properly validated the contents of $mycommand before executing!

Great writeup and a solution for problems like the one you described are definitely needed.
I myself didn't know about --% and don't remember ever seeing it unfortunately. Therefore the proposed &n operator might suffer from similar discoverability problems and it doesn't look natural to me to combine a symbol with a character (and it reminds me of %f in C printf statements). Since it's a breaking change anyway Is there a reason why it couldn't be for example &&?
Maybe it would me more intuitive to use something like brackets (or double braces?) to mark the area that you want to execute. Example: & [ wsl ls ] | less

I agree &! is the more intuitive way to go for those coming from other shells, however in super-duper-linter I am usually running native commands by building a parameter array and then splatting it to the command.

$command = 'linter'
$myargs = @(
    '-config'
    'linterconfig.path'
)
if ($Test) {$myargs += 'test'}
& $command @myargs

So this is the best way in case of conditional and whatnot for me

@bergmeister I originally thought of && as visually it's similar to & which many people know today. However, && being a pipeline chain operator may cause confusion about what it is supposed to do. I like the &! suggestion currently.

@JustinGrote, there's no reason to not continue to use that. the &! syntax is really for cases where you just want to cut and paste or have some args that conflict with PowerShell syntax and don't want to or don't know how to escape properly

Oh man, I remember having discussions with Bruce and Jason about this a decade ago. My gut feeling is that inventing another operator seems unnecessary here. I know that some people on this thread have not heard about --% but I'll wager than there are more than you think. What's wrong with:

# run ls in wsl, return results to powershell, then pipe to wsl again, to grep.
& wsl ls | wsl grep -i "foo"  

#  the entire pipeline right of wsl will run in wsl. 
& --% wsl ls | grep -i "foo"

Why hasn't anyone suggested this? It's logically consistent with usage of --% elsewhere to say "everything after this is to be passed without special powershell treatment."

@oising well for one, that command doesn't work because you have to preface it with the command you want to run, are you suggesting adding the functionality?

This kinda does what is expected:
function Invoke-LiteralCommand ($command) {Invoke-Expression "& $command --% $args"}

Invoke-LiteralCommand ping -W 200 www.google.com | grep icmp

@JustinGrote Yes, I know it doesn't work now :) I am suggesting that _instead_ of &n or &! or whatever else is being talked about. It just makes more sense to me: & is call and --% is suppress powershell parsing; together they are coherent and more discoverable than adding something new. I don't like the idea of having a "call" and a "call native" operator.

I've expanded my example to show the differences.

@oising I'd be cool with that over a new operator, though that is a lot of obtuse typing for "new powershell user who knows bash" which is what I assume this would be meant for.

@oising I'd be cool with that over a new operator, though that is a lot of obtuse typing for "new powershell user who knows bash" which is what I assume this would be meant for.

Not at all. It's for the PowerShell user that doesn't understand the complex quoting rules between PowerShell/cmd.exe/bash. As the issue title says "call native", for calling native executables.

@oising I'd be cool with that over a new operator, though that is a lot of obtuse typing for "new powershell user who knows bash" which is what I assume this would be meant for.

Not at all. It's for the PowerShell user that doesn't understand the complex quoting rules between PowerShell/cmd.exe/bash. As the issue title says "call native", for calling native executables.

True, or for those of us who'd rather not have to think about them at all.

Yeah I'm with @oising. This concept exists, it's just woefully insufficient. If we're going to implement something completely new we're better off deprecating/removing the old syntax.

I feel oftentimes these ideas are voiced, not enough weight is given to their actual target demographic. If users are _already_ struggling to find the existing methods, and finding the existing methods lacking when they are found, it's a call to a) improve documentation, and b) improve the actual functionality of the existing operators.

Instead we get a weird option c) which is supposed to somehow address the past issues whilst introducing yet _another_ operator which is relying on documentation which is already not accessible enough for users to find it naturally.

I agree that error messages and/or the suggestions system should be used to help introduce these concepts. I don't agree that we need a third? fourth? fifth? (I've literally lost count, someone help me out here) way to invoke commands. The existing methods are insufficient -- that means we should _improve_ them, not leave them behind like cobwebs to confuse users further when they start digging into them.

The first question I tend to get when people realise there's 5 ways of doing something is "why are there 2-3 ways that literally do the same thing but don't work as well", and my answer is as it always has been -- the PS team is _way_ too focused on making sure everything from the past decade++ still works, when trying to add/improve functionality. Here we see it again, we have existing but insufficient implementations that need revision and improvement, and the solution proposed is to further muddy the pool by adding another potentially incomplete implementation.

We should be finishing what we start, not starting another implementation all over again, IMO.

I believe if we are going to be using & for more than a few Unique use cases we should start thinking about making this considered a verb or look at standardization across the language.

The last thing I would want is confusion about what & is meant to do. In its entirety.

I'd consider 3 scenarios:

1. Full expansion - current PowerShell behavior.
2. Full literal - that is proposed in the OP.
3. Partial expansion - wsl ls $path still works

I wonder where @mklement0 comments? :-)

At start of reading this thread I thought; great idea! But after reading comments it started me thinking. I have always found the & command 'not PS standards worthy" and something reminding me of 'PERL days' (ugh). Introducing another non-standard PS command (not being noun-verb) will not help. Certainly not those less savvy at PS.

I did not know about the --% parameter either. I use the same principle as @JustinGrote as a solution.

Cutting/pasting commands into PS shell has never been my main 'thing'.
Would we not be better of replacing those native commands with PowerShell one's?
Make path for replacing cmd.exe entirely...

I vote for exploring improving existing commands. And - indeed - improving some documentation on --% and & usage

@iSazonov the use case @SteveL-MSFT is proposing is the 'cut paste' scenario. Partial expansion would make things much more difficult I think (on the AST side of things)

Immediately though of “#!” and then “!#”. Liking the “&!”.

We can probably look at adding token accelerators to function calls in the tokenizer. This would allow for the use of the following.

Iex -l
Or Invoke-Expression -literal that stops parsing before passing up.

I feel adding more unique tokens at the parsing level makes the barrier to entry go up and discoverability for this feature goes down.

Get-Help & for example doesn't show up with anything. And we have to look for it in about_Operators.

However there are unique cases for the call_operator that doesn't get documented.
& modulename { command }
allows you to scope into a module. And I am certain there are more. But the discoverability on it is so low that it becomes difficult to find within the native documentation. And the community knows about it only through JSnover and a talk he gave showing us cool things.

I believe whatever is decided on it needs to fully take into account how discoverable this "feature" is, from a new user perspective and keep in mind that newer users will try to use this on Powershell 5 not without knowing its new to pwsh core if the discoverability is too low.

I'd love us to change the existing behaviour, but changing such a fundamental API in PowerShell would just break so many things. We could possibly consider migrating over with a configuration.

Breaking things more directly might once have been possible before PS 6 went GA, but honestly, even then it would have been a serious threat to backward compatibility (not unlike Python's print() function).

In terms of passing arguments to subprocesses, I think both synchronous invocation and Start-Process invocation already have storied issues discussing an overhaul, and my feeling is that both need investing in at once for consistency. Start-Process in particular needs to update its support of passing an array of arguments.

For synchronous invocation with cmdline arg passing, I see four kinds of argument passing possible:

• Current behaviour, which would become legacy, which is subject to CommandLineToArgvW on Windows and has unexpected results
• Default behaviour, that should pass arguments by their expression value but apply the usual bareword token rules, so that things like > and | separate commands. In this mode, the value a subprocess takes from the expression should be what is displayed by Write-Host $val
• Verbatim behaviour, where all tokens are interpreted as bareword strings until the escape token is seen. This is what --% is supposed to do today, but ideally would have only one end token that can be embedded on the same line like --% ... %--
• Bash readline behaviour, where an alternate, sh-oriented escaping logic is applied, allowing simpler compatibility

I think the first behaviour should be slowly phased out by introducing the second as an experimental feature and then swapping them.

The second and third behaviours could have their own sigils like &! and &# or perhaps +% ... -% or something. But for the verbatim mode, I think an important facet would be simplifying the escape token so that more tokens are taken verbatim. Similar to a heredoc.

If the request is only to address copy-paste scenario like "Any text content after this operator will call into the "default shell" of the OS to execute." why do not say this explicitly by shell?
```powerhell
PS> shell wsl ls | less
PS> (shell wsl ls *.txt) | Select-String Hello

It is more discoverable and readable than cryptic operators in Forth/APL style.

I absolutely don't see how this would resolve #1995: See https://github.com/PowerShell/PowerShell/issues/1995#issuecomment-653174261

Also, the problem with WSL is a problem of the behavior of wsl.exe (see https://github.com/PowerShell/PowerShell/issues/12975#issuecomment-650353021) not of powershell.
(Ok, there is a problem with powershell, but that's #1995)

Oh, and isn't this almost a duplicate of https://github.com/PowerShell/PowerShell/issues/12975 ? Because I can essentially mostly repeat, what I mentioned there:

@bitcrazed

The problem is that the lack of an ability to delimit a portion of a command-line to be passed varbatim to the receiving command/script is something that trips users up all the time.

What do you mean by "portion of a command-line to be passed varbatim"? Do you mean 1. pass some sequence of characters verbatim to the called executable, such that the called executable has this sequence of characters in one element of its argument array (e.g. those characters are then available in `argv[1]` in [main](https://docs.microsoft.com/en-us/cpp/cpp/main-function-command-line-args?view=vs-2019#command-line-arguments)) Or do you mean 2. insert some sequence of characters verbatim into the [`lpCommandLine` parameter of `CreateProcess`](https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createprocessw#parameters) --- If you mean (1.), then it essentially already works:

PS /home/User> /bin/echo 'cd / && ls . | cowsay'
cd / && ls . | cowsay
PS /home/User>

(Except for the problem of embedded quotes as discussed in https://github.com/PowerShell/PowerShell/issues/1995) One could argue, that adding a one-line-here-string would improve some usecases, but I think, that's not really the point of this issue. As this does already work more or less, I assume, you meant (2.) --- If you mean (2.), then let me state my opinion on that in a somewhat dramatic way: Please please please don't add special syntax for this. This is basically what `--%` tried to do, which also should have never ever been implemented. Why am I so strongly against this? 1. It is a Windows only problem, so adding syntax would mean that powershell on Windows has different syntax than on Linux. (Or the syntax would be supported but is totally meaningless, as it is currently the case for `--%`) 2. If the main commandline shell on Windows published by Microsoft adds a first-class feature (via special syntax opposed to via a cmdlet) to call executables that don't follow the typical [commandline parsing rules](https://docs.microsoft.com/en-us/cpp/cpp/main-function-command-line-args?view=vs-2019#parsing-c-command-line-arguments) (if the tool follows the typical rules, you don't need (2.), you can usually better use (1.)), then that encourages authors of command line tool, to not follow these rules, which only worsens the ["Windows command line anarchy"](https://stackoverflow.com/a/4094897/2770331). The less people follow the typical rules, the harder it is to programmaticly call external executables or generally write cross platform code, so I definitely think, that program authors should be encouraged to follow those typical rules. 3. I strongly doubt, that this is a common use-case >And this isn't just an issue that affects WSL: It also affects Windows Terminal's wt command-line invocations, and many other tools. Could you add some examples, where such problems occur? Because in case of WSL, I'd say that WSL's parsing of the commandline is simply [broken](https://github.com/Microsoft/WSL/issues/1746) (issue was about `bash.exe` but situation is by default not better with `wsl.exe`) in the default case - I'd consider every tool, that doesn't follow the typical [commandline parsing rules](https://docs.microsoft.com/en-us/cpp/cpp/main-function-command-line-args?view=vs-2019#parsing-c-command-line-arguments) broken, but WSL's default behavior is IMHO not even properly documented... I said the "default" behavior of `wsl.exe` is broken - while writing this response, I noticed, that `wsl.exe` actually seems to behave as expected, when using `-e`:

PS C:\> wsl -e bash -c 'cd / && ls . | cowsay'
 _______________________________________
/ acct bin boot cache cygdrive data dev \
| etc home init lib lib64 lost+found    |
| media mnt opt proc root run sbin snap |
\ srv sys tmp usr var                   /
 ---------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||
PS C:\>

So the only thing missing for this explicit usecase is IMO a parameter to `wsl.exe` to call the default shell with the commandline arguments parsed as in any other normal exe.

The only thing regarding "portion of a command-line to be passed varbatim to the receiving command/script", that would be improved with this operator would be meaning (2.) above, and as mentioned, I think developing in that direction is bad.

If all you want to do is a shortcut do copy&paste existing command lines, why don't you add a cmdlet (e.g. Invoke-Shell, that calls bash -c on linux and cmd /c on windows?
Then you could just do

Invoke-Shell @'
whatever existing comandline containg '"quotes"' or whatnot
'@

If it must be one line, then add one-line-here-string syntax, but please don't implement a special operator, that can't be properly found and just complicates everything.

And please don't abandon #1995 for this!

+10 points for using the word "sigil".

The thing that is nice about &! versus shell is that "shell" is a common English word, therefore people might have an existing command named shell. (In fact, I have both a function named "shell" and a function named "n" (hence I don't like &n).)

@oising: & --% is nice, except one of the express purposes of the proposed feature is to mean that | becomes something that is passed to the native shell; not something that brings you back into PowerShell parsing. And if we make the pipeline precedence different for & --% than for foo $blah --% more args, I would find that bewildering... what if you did & $foo --% more args | blah? Thus in my opinion, the proposed &! is different enough that it warrants having a different operator. (Put another way: the proposed functionality is related but different than combining & and --%.)

This feature is more than just for "new PowerShell user who knows bash" and "user that doesn't know all the complex quoting/escaping rules". It's also for the most expert of users, who just want to copy/paste a command from StackOverflow without having to go in and fiddle with everything. This happens to me quite a bit--a product I work on has a lot of documentation about how to do things, and the lingua franca is cmd.exe commands--so I have to go and deal with fixing the %VARIABLE_REFERENCES% etc.; it would be so much nicer to type &! and then paste. In fact if this is implemented I will probably refer to it as "the StackOverflow operator". :D

@vexx32: This isn't the "one ring to rule them all" call operator; just another tool in the toolbox to address a common problem. I don't think we should get rid of existing tools to make room for this one; they do different things. I understand that new users may feel overwhelmed when the toolbox has a hammer, a screwdriver, a drill, an impact drill, and a nailgun... but despite the fact that they are all for "poking a pokey thing through one or more other things", they are invaluable for a professional; all used in different (though related) situations.

@TSlivede: I agree that #1995 is separate. But I disagree with "It is a Windows only problem": copying and pasting platform-specific commands from StackOverflow applies equally to any OS. (In fact, this would be very helpful for many people like me who are stronger on one platform than another--I am more proficient with Windows technologies, so when on Linux I tend to do a LOT of copying and pasting from SO.)

Whether via a new operator &! or some Invoke-Shell command, I do not have such strong feelings as others... I sympathize with people complaining that the operators are hard to search for, etc. but I really like the terseness of &!.

@jazzdelightsme this isn't "another tool in the toolbox" as much as it is "another incredibly specific screwdriver attachment for a screwdriver that already has about 3-5 that are currently already _okay_ and could stand to be improved" in my opinion.

@jazzdelightsme The "Windows only" part was perhaps more related to the issue where I originally posted that statement - yes OK, this is not an exact duplicate of that issue.

If #1995 is not abandoned and the feature requested here is not resolved via special Syntax but via a commandlet (or perhaps new Syntax for a one-line here string that works everywhere not just for this special case) then I am actually in favor of adding this functionality. (Or maybe not "in favor" but "This is totally acceptable to me")

Another idea: If this is issue is actually about copy&pasting full commands that were intended for native shells, maybe PsReadLine could add a feature to roughly translate a command from bash or cmd style to powershell. The translation could be triggered by a special key combination or automatically if a parsing error occurs when trying to execute a command.

This would have the additional advantage of teaching users the Syntax of powershell.

Regarding discoverability: PsReadLine could always display a message for a short time that the specific key combination can be used, whenever syntactically incorrect content (that would be valid after translation) is pasted.

But I could understand if this was just a bit to complex to realize.

@jazzdelightsme did you talk to @joeyaiello? I thought only he called it the StackOverflow operator which made me cringe even though it's very true

@SteveL-MSFT No I didn't talk to anybody about this; I arrived at "StackOverflow operator" independently. xD

@jazzdelightsme

... except one of the express purposes of the proposed feature is to mean that | becomes something that is passed to the native shell; not something that brings you back into PowerShell parsing.

I'm not sure you've understood my suggestion; what you say is _precisely_ what I'm saying. Let me be more verbose with some strategically capitalized hints :D

# NON-OPTIMAL SOLUTION: run "ls" in wsl, return results to powershell, then pipe to wsl again, to "grep."
& wsl ls | wsl grep -i "foo"  

# NEW SOLUTION: the entire pipeline/string, INCLUDING THE PIPE SYMBOL, right of wsl will be executed in wsl. 
& --% wsl ls | grep -i "foo"

# NEW SOLUTION: the entire pipeline AS ABOVE is sent to WSL, then the results are returned to powershell
# and piped, in powershell, to foreach-object
(& --% wsl ls | grep -i "foo") | % { "match {0}" -f $_ }

# NEW SOLUTION: allow placing --% beyond first argument
# pass everything right of --% to the command in $wsl 
$wsl = gcm wsl
& $wsl --% ls | grep -i "foo"

# or

& wsl --% ls | grep -i "foo"

# NEW SOLUTION: execute multiline pasted code without powershell parsing.
& --% @'
(pasted multiline code)
@'

What am I missing here? If the pipe appears after --% it's just another dumb character: an argument. There's no twiddling with operator precedence. If contained in a nested pipeline or subexpression, then start parsing for powershell again.

what if you did & $foo --% more args | blah?

Well, that would attempt to execute the argument in $foo and pass "more", "args", "|" and "blah" to be treated as whatever's in $foo wants. The rules seem simple enough to me, but admittedly I've been using powershell/monad a long time. This isn't a major breaking change to & as & $foo --% ... treats --% like any other argument. So unless we've been using this sigil elsewhere nefariously, we should be safe.

Thoughts?

I made multiple edits to the above, to cover partial line evaluation, multiline evaluation and variable positioning of --%. All without introducing new sigils, operators or strange syntax. I think that covers it. The only thing is: Can the parser handle it? What do you think @lzybkr @JamesWTruher @BrucePay ?

Hear, hear, @TSlivede.

Forgive me for being blunt, but we've been talking about these issues for a looooong time:

What this issue proposes amounts to magical thinking:
It is related to the ill-fated, inherently problematic "stop-parsing symbol", --%, which in its current form is of limited use _on Windows_ (see https://github.com/MicrosoftDocs/PowerShell-Docs/issues/6149) and virtually useless on _Unix-like platforms_ (see https://github.com/MicrosoftDocs/PowerShell-Docs/issues/4963).
--% should never have been implemented, and neither should this proposal be implemented as presented.

The magical thinking is a conceptually muddled conflation of two ideas:

• (a) "I want to pass space-separated tokens to some external program, allowing any barewords, without needing to worry about shell metacharacters such as & or |." - see https://github.com/PowerShell/PowerShell/issues/12975#issuecomment-646276628

  • Of course, such a feature is at odds with using such commands _as part of a pipeline_ or with _pipeline-chaining operators_ (&& and ||), and also placing additional commands on the same line, after ; - and, not least, being able to use any _PowerShell variables and expressions_ in the command line.

• (b) "I want my command line treated the way I'm used to _from the shell I normally use_ (or the shell for which the command line was originally written)", which, of course, _is_ subject to unquoted use of characters such as & or | being interpreted as metacharacters and therefore having _syntactic_ meaning.

  • Case in point: neither cmd nor /bin/sh (or any POSIX-like shell such as bash) would accept wsl ls | less in a way that passes ls, | , and less _verbatim_ through to wsl - they'd all interpret _unquoted_ | as _their_ pipe symbol.

These ideas are at odds at one another, and require distinct solutions - neither of which requires special syntax per se:

• The solution to (a) is to _abandon_ the idea and to instead use _PowerShell's own, existing syntax_ to ensure that metacharacters are used _verbatim_ - which invariably necessitates _quoting_ or _escaping_,:

  • Either: quote barewords (tokens not enclosed in quotes) that contain metacharacters _as a whole_: wsl ls '|' less
  • Or: `-escape metacharacters _individually_ in barewords: wsl ls `| less
  • Or: use an _array [variable]_ to pass the arguments _verbatim_: $wslArgs = 'ls', '|', 'less'; wsl $wslArgs
  • Of course, #1995 demonstrates that, regrettably, this approach is and always has been broken when it comes to arguments that have _embedded_ " _when calling external programs_ - _this needs to be addressed_: see below.

• The solution to (b) is to actually _call the shell I normally use_, passing the command line _as a single string_.

  • As @TSlivede suggests, an Invoke-Shell cmdlet to which you pass the command line _as a single string_ and which calls the _platform-native shell_ is the solution - and we already have syntax for verbatim strings ('...' or its here-string variant, @'<newline>...<newline>'@
  • My recommendation is to call it Invoke-NativeShell, and to call /bin/sh - not bash - on Unix-like platforms.
    
    
    
    • Additionally, I suggest defining ins as a succinct alias.
    
    
  • This approach solves all conceptual problems with --% and with the proposal at hand:
    
    
    
    • It make it very obvious where the command passed to the native shell ends (by virtue of being _a single string_), and it allows using an Invoke-NativeShell / ins call in a regular PowerShell pipeline / with PowerShell redirections without ambiguity.
    
    
    • You can optionally use an _expandable_ string ("...." or its here-string variant) in order to embed _PowerShell_ variable values in the command line (something that --% can't do).
    
    

Since the powers that be have concluded that #1995 cannot be fixed lest backward compatibility be broken - after all, all existing workarounds would break, see https://github.com/PowerShell/PowerShell/issues/1995#issuecomment-562334606 - the best thing we can do is to support the opt-in into the proper behavior with a syntax that is as succinct as possible, so as to minimize the pain of having to go out of your way to get the behavior that should always have been the default.

Therefore: special syntax such as &! should only be provided as the opt-in to get proper passing of arguments to external programs in order to address #1995, based on the behavior proposed in @TSlivede's RFC - see https://github.com/PowerShell/PowerShell-RFC/pull/90. If a future PowerShell version is ever permitted to break backward compatibility, then it is direct invocation / invocation with & that should exhibit this behavior.

From a user's perspective, this means: All I need to do is to satisfy _PowerShell's_ syntax requirements - see https://github.com/PowerShell/PowerShell/issues/1995#issuecomment-640711192 - and I can rely on PowerShell to robustly pass the resulting verbatim tokens through to the target program - _this is a core mandate of any shell, which PowerShell has regrettably never satisfied with respect to external programs_.

For instance, the following command - which currently does _not_ work as intended (when invoked directly or with & - see this docs issue) - should then work:

PS> &! bash -c 'echo "one two"'
one two   # Currently (invoked with & instead of &! or with neither) prints only 'one', 
          # because PowerShell passes the entire argument (brokenly) as  "echo "one two"",
          # which the target program sees as *2* verbatim arguments `echo one` and `two`

Those looking for a workaround in current / earlier PowerShell versions can find a simple function that will cover most use cases at https://github.com/PowerShell/PowerShell/issues/1995#issuecomment-303345059 (limitations discussed at https://github.com/PowerShell/PowerShell/issues/1995#issuecomment-649560892), and a more elaborate version at https://github.com/PowerShell/PowerShell/issues/1995#issuecomment-562334606

P.S.: The discovery problem of (not only symbol-based) operators is a separate problem that is well worth addressing in its own right: the solution is to _enhance Get-Help to allow operator lookup_ - see #11339, which also links to an interim solution.

Since the powers that be have concluded that #1995 cannot be fixed lest backward compatibility be broken - after all, all existing workarounds would break, see #1995 (comment) - the best thing we can do is to support the opt-in into the proper behavior with a syntax that is as succinct as possible, so as to minimize the pain of having to go out of your way to get the behavior that should always have been the default.

@mklement0 I think opt-in was a part of https://github.com/PowerShell/PowerShell/issues/1995 and that was rejected. So to me it sounds like wishful thinking. Am I wrong?

@AndrewSav, we originally only discussed an opt-in via configuration or preference variable.

However, @joeyaiello said the following in https://github.com/PowerShell/PowerShell/issues/1995#issuecomment-653164987 when he closed the issue (emphasis added):

Anything we touch here is just too much of a breaking change, and we should address the problem with a new operator

A new operator _is_ an opt-in of sorts, but an unproblematic one, I hope (though the need for it is unfortunate, but that's the price of backward compatibility).

While this proposal purportedly also solves #1995, it doesn't, given that it aims (primarily) at accommodating the syntax of _other_ shells. Given that both problems are worth solving, independently, my proposal was this:

• Let's introduce a new operator, but use it solely to fix #1995 (via https://github.com/PowerShell/PowerShell-RFC/pull/90), i.e. to fix PowerShell's _own_ (non-other-shell-emulating) broken handling of argument passing to external programs.

  • New code should then always use &! (or whatever form we decide on) instead of & when calling external programs (you can think of it as deprecating & for external programs).
  • When calling _PowerShell_ commands, &! should act like & does now (there never was a problem), thereby enabling the consistent use of &!.

• To solve the reuse-command-lines-from-other-shells problem, let's implement a Invoke-NativeShell / ins cmdlet that calls the platform-appropriate native shell (cmd /c on Windows, /bin/sh -c on Unix-like platforms) with the command line passed as a _single string_.

  • In the simplest case, with no ' present in the command line, a regular single-quoted string will work; e.g.:

    ins 'ls | cat -n'

  • With ' present, they can either be _doubled_; e.g.:

    ins 'echo ''hi'' | cat -n'

  • or, if the desire is not to have to touch the command line at all, a here-string can be used (as already shown by @TSlivede) - while this syntax is a little more cumbersome, it should always work:

    ins @'
    echo 'hi' | cat -n
    '@

  • additionally, by using an _expandable_ (interpolating) string ("..." or @"<newline>...<newline>"@), you have the option of incorporating _PowerShell_ variables into the string passed to the native shell (something that --% doesn't support):

    $foo = 'hi'
    ins @"
    echo '$foo' | cat -n
    "@

  • Note:

    • Invoke-NativeShell cmdlet would be a fairly thin wrapper around cmd /c / /bin/sh -c calls, which you could obviously make directly yourself, but it does have the advantage that you don't need to know the specific executable name and command-line passing syntax.
    • As a potentially beneficial side effect, you'll be able to use common parameters such as -OutVariable, and can think about whether to let the -Error* parameters act on _stderr_ (though, for symmetry, Invoke-Command should then too, which would be a breaking change).
      
      
      
      • Since POSIX-like shells support _multiple_ arguments with -c - the first constituting the code to execute, and the subsequent ones the arguments to pass to that code, starting with $0(!) - e.g. bash -c 'echo $@' _ 1 2 3 - optional additional arguments must be supported too.
      
      
      • Since the platform-native shell is being called, such calls are by definition _not_ portable, due to the fundamental differences between cmd and sh.
        
        This is why it's so important to fix PowerShell's _own_ argument-passing (#1995), as it then provides a reliable, truly cross-platform method of invoking external programs.
      
      

The final piece of the puzzle is _documentation_.

https://github.com/MicrosoftDocs/PowerShell-Docs/issues/5152 is a longstanding proposal to write a conceptual help topic about calling external programs (e.g., about_Native_Calls), which should, in a single location, cover _all_ aspects that are relevant to calling external programs; aside from what's already covered there (syntax pitfalls due to additional metacharacters, absence of a raw byte pipeline, non-integration with PowerShell error handling, ...), the issues at hand should be discussed as well:

• How passing arguments with embedded " (and _empty_ arguments) is broken in (hopefully soon-to-be) earlier versions (currently discussed in a separate proposal, https://github.com/MicrosoftDocs/PowerShell-Docs/issues/2361), and how to work around that (manual \-escaping or use of one of the helper functions from #1995, one of which is succinct enough to be directly included in the topic).

• How only &! should be used going forward to call native (external) executables, so as to ensure proper argument-passing.

• How Invoke-NativeShell / ins can be used to execute command lines _written for the platform-native shell_ as-is.

Interesting, today users can:

1. direct invocation
2. & operator
3. Start-Process

It is not at all clear that users should use.

After fixing broken behavior users will have the following features:

1. direct invocation
2. & operator
3. Start-Process
4. &! operator
5. Start-ProcessV2

Is not it too much for the shell to run ping 127.0.0.1?

Yeah that's my main concern with adding yet another operator as well @iSazonov. It's already confusing, and this will not help. Documentation is only a part of the puzzle. Even if we have decent documentation, the mere fact that there are _five different ways_ to do it, each of which behaves a bit differently, will always cause confusion.

If we care about the UX at all, we should be opting to improve existing cases over adding yet more.

If we care about the UX at all, we should be opting to improve existing cases over adding yet more.

Which you cannot because you must be backward compatible. I feel we are going in circles. https://github.com/PowerShell/PowerShell/issues/1995 was that and it was closed for that very reason.

@iSazonov:

The existing lack of guidance needs to be addressed even if we don't introduce a new operator:

• comprehensively, via the proposed about_Native_Calls topic

• in the Start-Process topic specifically, which currently lacks any guidance as to when it use is appropriate - see https://github.com/MicrosoftDocs/PowerShell-Docs/issues/6239

To summarize briefly:

• Don't use Start-Process to invoke _console_ (terminal) programs (unless - which is available on Windows only - you want to run them _in a new window_).

• Direct invocation and invocation via & are _equivalent_; (leaving script blocks aside),& is only ever needed for _syntactic_ reasons, whenever the command name or path is _quoted_ and/or contains _variable references_.

After fixing broken behavior users will have the following features:
direct invocation

• & operator
• Start-Process
• &! operator
• Start-ProcessV2

• No, there won't be a Start-ProcessV2, just a _new parameter_, -IndividualArguments - see https://github.com/PowerShell/PowerShell/issues/5576#issuecomment-552124719

• Yes, there will be &! (or whatever symbol we decide on), but it effectively _supersedes both & and direct invocation_ (at least for _external programs_, but you'll be able to use it with _all_ command forms) - which should be easy to communicate.

Is not it too much for the shell to run ping 127.0.0.1?

ping 127.0.0.1 will continue to work, but if you want bash -c 'echo "one two"' to work, you'll have to use
&! bash -c 'echo "one two"'

That is unfortunate, but - given that the direct invocation / & behavior _cannot be fixed_ - it strikes me as the best solution.

Fortunately, it's a simple rule to memorize: _If you want predictable argument passing to external programs, use &!_

Alternative forms of opt-in - configuration (via powershell.config.json) or a preference variable - are much more problematic:

• The dynamic scoping of preference variables makes it too easy to inadvertently apply the opt-in to legacy code that wasn't designed for it - see https://github.com/PowerShell/PowerShell/issues/1995#issuecomment-650262164

• Configuration would apply to sessions as a whole, making opt-out impossible altogether.

• In both cases, action is needed that is _separate_ from a specific call, and looking at a given call won't tell you how it will behave - by contrast, &! leaves no doubt as to which behavior will ensue.

To the end user, it looks just as awful as if we launched a calculator and found there 5 buttons for the addition operation. 😄

If we want to improve it so much that we are ready for such complications, then perhaps this is enough to accept a breaking change (for direct invocation and &) so as to simplify the behavior and release V8.0.

If our desire is not so strong, it is possible to sufficiently improve the diagnostics (prog.exe param1 param2 -Whatif and the same for Start-Process -WhatIf) so that it displays what testargs program displays and recommendations on how to correctly perform escaping.

The script developer can afford to spend time testing and using the debugger. But in an interactive session, any user wants simplicity, brevity and clarity, and this outweighs everything.

@iSazonov:

then perhaps this is enough to accept a breaking change (for direct invocation and &) so as to simplify the behavior and release V8.0

You certainly have _my_ vote - but do the powers that be support this plan? [_update_ - see #13129]
If we do take a change of this magnitude, we can finally tackle all historical baggage - see #6745

If we want to improve it so much

There are two good reasons to want it that much:

• To end years of pain due to broken argument-passing - as evidenced by #1995, many related issues, and countless Stack Overflow questions.

• Because - as has been stated several times before - calling external programs with arguments is a _core mandate of any shell_.

Historically, in the Windows-only days, the dearth of capable external CLIs made the inability to call them properly less problematic, and users who stayed in the walled garden of PowerShell (calling only PowerShell-native commands) didn't feel the pain. But times sure have changed:

We live in the age of a wealth of cross-platform CLIs that are crucial to development, CI, and deployment tasks.
If PowerShell wants to be taken seriously as a shell, this problem must be addressed.

Even if an allowed-to-break-backward-compatibility v8.0 happens (fingers crossed), its release could be a long time away, and we _also_ need an interim solution.

As much as the proposed enhancement to the documentation are needed, I don't think they alone sufficiently ease the pain, for lack of a readily available runtime solution.

I get that the operator is ugly, but a non-breaking solution _necessitates_ additional syntax (more to type), whatever its form.

An alternative to &! that is less ugly, easier to type, and doesn't "pollute" the language is to ship a built-in _function_ with a concise name, such as the iep function (Invoke-ExternalProgram) that masks all the problems, shown here.
(Note that we already ship functions that either wrap other functionality (e.g., pause) or accommodate cmd.exe users (e.g., cd.. [sic]).)

Thus, to get bash -c 'echo "one two"' to work properly, you'd call iep bash -c 'echo "one two"'
Or, to use perhaps a more realistic scenario:

# Pass JSON to an external utility (Unix example)
# Without `iep`, the embedded double quotes are lost and *broken JSON* is passed.
iep /bin/echo '{ "foo": "bar" }'

Again, we're ultimately talking about a _compromise_ in service to preserving backward compatibility - this is inevitably a nuisance to end users.

improve the diagnostics (prog.exe param1 param2 -Whatif and the same for Start-Process -WhatIf) so that it displays what testargs program displays and recommendations on how to correctly perform escaping.

Since direct / &-based invocation isn't a cmdlet call and all arguments are presumed to be pass-through arguments, treating -WhatIf as a common parameter amounts to a breaking change (however unlikely that is in practice).

If you're willing to accept such a change, we might as well (also) support a -DoItRight switch (or a symbol-based one, akin to --%) as an opt-in for the fixed behavior - and that strikes me as worse than the iep-prefix / &! approach.

(A non-breaking diagnostic option, I suppose, would be to enhance Invoke-Command, which currently doesn't support -WhatIf - however, if we provide a well-documented opt-in to the correct behavior, I don't think this is worth doing.)

I agree with @iSazonov that the situation is not ideal to introduce yet another thing to do something that it supposed to work, but as mentioned by @mklement0 waiting for a breaking release that properly fixes this could be a very very very long time, and a lot of us desperately need a solution ASAP. For all I can see, the best solution currently is to just use
https://github.com/PowerShell/PowerShell/issues/1995#issuecomment-562334606 right?

@mklement0 I'm not sure how a iep /invoke-externalprogram could work, since a cmdlet is subject to the same parameter munging we're trying to fix. This is a parser issue, right? If we tried to hide everything inside a cmdlet/function, I'd imagine it would be hideously complex given it would have to undo the munging/reconstruct intent. Or am I missing something?

@oising:

To be clear: the linked iep function fixes #1995, not what _this_ proposal is primarily about (which I think is _not_ worth pursing, as argued above), though do note that the stated intent in the OP is to _also_ solve #1995.

1995 is about being able to rely on _PowerShell_'s syntax (not another shell's) for passing arguments to external programs _faithfully, as seen verbatim by PowerShell after its own parsing_.

PowerShell's own parsing is fine, and there are no problems as long as you pass arguments to _PowerShell_ commands.

By contrast, passing arguments to _external programs_ is badly broken, and that's what iep fixes.

It is broken due to flawed _re-quoting and escaping_ of the arguments _behind the scenes_, when the command line ultimately used is constructed.
https://github.com/PowerShell/PowerShell-RFC/pull/90 proposes a proper fix (sans agreement on how to handle backward compatibility, and with a few minor questions yet to be answered), that at its core relies on the recently introduced System.Diagnostics.ProcessStartInfo.ArgumentList to perform the proper re-quoting and escaping for us (we just pass the collection of arguments that resulted from PowerShell's own parsing) - which, incidentally, is only ever required _on Windows_; on Unix, sensibly, _there is no command line_ (outside of a shell) when you pass arguments on process startup: you just pass the arguments verbatim, as an array.

In short:

• Providing iep as a built-in function would be a low-ceremony stopgap opt-in to offer proper argument passing to external programs, until a proper fix - in the context of a breaking change - can be implemented

• An Invoke-NativeShell (ins) cmdlet is the proper solution - not a stopgap - to providing a way to call _a different shell's_ (the native shell's) command lines, using _its_ syntax - see above.

Currently, there are cases where cutting and pasting a PERL command fails to run as expected in Python. Go figure. This must be a great disadvantage for all those cutters-and-pasters out there. I think Python should have a special operator for that.

I'm not sure you've understood my suggestion; what you say is _precisely_ what I'm saying. Let me be more verbose with some strategically capitalized hints :D

# NEW SOLUTION: the entire pipeline AS ABOVE is sent to WSL, then the results are returned to powershell
# and piped, in powershell, to foreach-object
(& --% wsl ls | grep -i "foo") | % { "match {0}" -f $_ }

What am I missing here? If the pipe appears after --% it's just another dumb character: an argument. There's no twiddling with operator precedence. If contained in a nested pipeline or subexpression, then start parsing for powershell again.

Thoughts?

What you are missing is the need to pass a closing parenthesis as an argument to an external program. It is a typical DWIM approach—but PowerShell is not HAL.

Thanks for all the detailed feedback, everyone. Speaking as myself here. A few uber-points before I get more into the nitty gritty:

• I'm still not on board with a vNext/8.0 that decides to break a whole new litany of stuff. Backwards compatibility matters. I know it's painful, and I'd rather live in a world where we could go back and correct every mistake that PowerShell ever made, many unintentional. But vast majority of PS users are not hanging out here in issues, knee-deep in SO edge cases. They've copy/pasted the same workaround to something for 4-10 years, they don't necessarily know why it works, and they won't care that we were "fixing mistakes" when their script breaks on upgrade. They just know that "updating PS has broken me, and I can't trust updates", and then we end up with even more users slowing down their updates and adoption.
• I think some folks in this issue may be projecting many different hopes and desires into this issue, or see it as an either/or with other fixes. I realize I exacerbated that by closing #1995, and I'm sorry for that (though I will say, I'm still not sure that one specifically can actually be fixed, but we will take a harder look at it), but this is really a "trap door" solution that gets both new and experienced users out of a bind quickly without letting frustration mount. We can still have longer-term conversations about fixes to the PS parsing side of the house so that we can offer a truly cross-platform solution for everything.

Now, comments as I read:

• I'm wary about the proposed desire to align with #! (with either &! or $! given that, in the default case, users are likely not going to specify the interpreter. If I want to do something like $! net <stuff>, net.exe is not my interpreter. And on non-Windows, the "native" thing to do is still to pass to sh/bash/whatever before passing to the binary being run. E.g. I don't want Python parsing *.py in the string $! /usr/bin/python *.py or even $! python *.py. I want Bash to do the globbing and then to hand the list of matches to python.
• Like @oising suggested, I'm in favor of --% as the operator, overloaded to be used in the front. Discoverability is a problem, and given that we're already 10 years in on working on the discoverability of this one (it's Google-able now!, I say we just go with that. (I was the one that made @SteveL-MSFT put it in his proposal as an alternate consideration). Question, though: do we even need the & call operator here? Why not just start the line with --% native.exe do /some $stuff?
• @rjmholt: I didn't totally understand your comment here, but it's my take that we should probably not change anything with the existing --% behavior where it comes after the executable, but I think we can do your "new behavior" with where it comes before the executable.

• Read this from @jazzdelightsme and I emphatically agree:

  It's also for the most expert of users, who just want to copy/paste a command from StackOverflow without having to go in and fiddle with everything.

  In my mind, this is a huge aspect of the scenario, possibly the biggest one. It's not just SO, either, it's docs from around the web that pre-suppose you're in a Bash-like shell.

• Irrespective of whether things like #1995 gets fixed, the point there is that a) there's a long tail of weird edge cases like that, and b) they're really hard to fix without breaking people. It's not necessarily true of all them, but we've hit it enough over the last couple years that it's apparent to me we need a "trap door" like this to get people out of hairy situations where they don't want to dive down the rabbit hole of our nested escaping (only to discover there's a bug like #1995 that prevents them from working even once they do know what they're knowing).

In reviewing the comments, I think there are two open questions we still need to answer:

1. Do we continue to parse command separators and everything after them in PowerShell, or does the command separator also get passed verbatim to the native shell?
   I say that we should pass everything verbatim to the native shell as that's currently something that you can't do, and that @oising's solution of a leading parenthetical may be the solution in that we can still support pipes, pipeline chains, etc. if you want to mix/match native and PS parsing (though I want to get some opinions from those closer to the parser).
2. What's the operator?
   As alluded above, I like --% because it's already got a reputation as the "just do what it used to do" operator, and we're going to have to reset on discoverability with anything new. I understand arguments like @mklement0's that, if we do this, we are changing the technical meaning of --% (and @SteveL-MSFT has made this argument to me as well), but I don't think most people out in the wild are thinking of --% in terms of "it does what it used to do in cmd". I don't think it's a stretch to codify this assumption into the operator's definition as "do what the native shell does here".

Orthogonally, just thought of another scenario we may or may not want to support.

On Windows, cmd (effectively) includes the current directory in its PATH lookup. However, PowerShell requires that the user pre-pend a ./ in order to execute any EXEs (or anything in PATHEXT) if the current directory is not in PATH.

So my open question is: do we want to actually hand the entire command string to cmd? I'd propose that if we really want to light up the copy/paste scenario with this operator, that the answer is yes, as some third-party docs/tutorials don't necessarily detail how to put the tools they expect you to use into the PATH. It probably doesn't matter that much either way, but I did want to enumerate it here.

• I'm still not on board with a vNext/8.0 that decides to break a whole new litany of stuff.

Yeah I don't know how you fix these problems without breaking in incredibly difficult to troubleshoot ways. Plus I don't know about y'all but even I don't want to have to juggle all these complicated binder/parser differences when writing for multiple versions.

Usually if I'm arguing against breaking changes it's for other folks. This though, seems like it'd probably be a PITA to deal with even for folks present in ITT.

• Like @oising suggested, I'm in favor of --% as the operator, overloaded to be used in the front. (...) Why not just start the line with --% native.exe do /some $stuff?

I really like this. Ideally as close to calling CreateProcess/exec as possible (plus nix env var handling and in/out redirection or console attachment). Or maybe falling back to cmd.exe/bash completely.

I'm not stuck on keeping & in front of --% either. I can justify the idea of treating raw --% like a directive/operator in my head at least. What about multiline? Allow --% with @" and "@ for var substitutions and subexpressions, and single quote @' and '@ to pass literal? Without the here strings, it's single line only?

Like @oising suggested, I'm in favor of --% as the operator

Works for me.

Question, though: do we even need the & call operator here?

As long as I can still do --% & $computedPathToNativeExe foo;@workspace then fine by me. We do need to be able to use the call operator with this.

I say that we should pass everything verbatim to the native shell as that's currently something that you can't do

Agreed.

I don't think it's a stretch to codify this assumption into the operator's (--%) definition as "do what the native shell does here".

Also agreed!

Also, maybe this is a way to get in the feature to define env vars before the executable that are defined just for the executable's session (#3316):

--% JEKYLL_ENV=production jekyll build

Just a thought.

@rkeithhill, the idea with the call native operator is that the line isn't processed by PowerShell and sent verbatim to the "default shell" of the OS. So in this case, your first example --% & $computerPathToNativeExe foo;@workspace would not work. You would have to use the current method and escape as appropriate. The second case --% JEKYLL_ENV=productoin jekyll build should just work.

the line isn't processed by PowerShell and sent verbatim to the "default shell" of the OS

OK, that seems reasonable. However this --% JEKYLL_ENV=productoin jekyll build would just work on Linux/macOS but not on Windows - at least not without some help from PowerShell, right?

@rkeithhill as currently proposed that won't ever work on Windows as it relies on the OS "default" shell. For PowerShell declaring env var for a process, we have a different issue opened to discuss that.

@PowerShell/powershell-committee discussed this today, I've updated the original proposal to reflect the result of that discussion. We'll move forward with an experimental feature implementation and get more feedback from users with working code. We are still open on the sigil for the operator, but using --% for now.

where $foo is a literal filename and $PWD is current directory within WSL. Note that in the first example, you would have to know to escape && to have it execute within WSL instead of within PowerShell.

Both $foo and $PWD are variable references to be evaluated by sh.

To pipe output from such execution back into PowerShell, the user is required to store the results into a variable first:

$out = --% ls *.txt
$out | select-string hello

Note that unlike the current & call operator, you cannot use any PowerShell syntax, so:

--% $commandline

will try to execute $commandline literally (under cmd) or after shell expansion (under sh). PowerShell will not attempt to resolve that variable.

The cut & paste problem is solved by simply pasting after &n is typed.

by simply pasting after --%

The above example for wsl would look like:

Which one?

Since all of the arguments after --% is passed to the "default" shell, to support pipelining into PowerShell, you would need to use a variable:

This is a repetition.

If we are going to add something useful I propose a Pivot.

Why cant we have something like a herestring Literal

@!""!@

So we can have a real way to do this instead of a --% which also has its own host of issues.

Atleast that way we can bury the issue with "better" design tactics.
And discoverability for these edge cases to sit in About_Strings where it should be solved in.

Just a thought.

I fail to see how @!"…"!@ is better than Invoke-NativeShell @"…"@. Are we trying to make PowerShell more Brainfuck-like?

@yecril71pl

I fail to see how @!"…"!@ is better than Invoke-NativeShell @"…"@. Are we trying to make PowerShell more Brainfuck-like?

I believe, that @romero126 meant to suggest a oneline herestring literal, such that one could write

Invoke-NativeShell @!"complicated native command with "quotes" and | pipe"!@

instead of

Invoke-NativeShell @"
complicated native command with "quotes" and | pipe
"@

If such a oneline herestring would be considered, I'd suggest to use syntax like @"""…"""@ and @'''…'''@, this would be IMO easier to type than @!"…"!@.

@romero126 Could you clarify what you meant?

@yecril71pl thanks for catching those errors, updated the top message with fixes.

It is unspecified whether --% will execute as command line or as a batch file.

I believe, that @romero126 meant to suggest a oneline herestring literal, such that one could write

Invoke-NativeShell @!"complicated native command with "quotes" and | pipe"!@

instead of

Invoke-NativeShell @"
complicated native command with "quotes" and | pipe
"@

If such a oneline herestring would be considered, I'd suggest to use syntax like @"""…"""@ and @'''…'''@, this would be IMO easier to type than @!"…"!@.

@romero126 Could you clarify what you meant?

I believe if we add flexibility to Strings to handle "almost raw" unparsed data we wouldn't have issues about this pop up.
@TSlivede I agree almost Completely with what you said. Only with a slight variation. It should be able to handle BOTH single line and Multiline without interpolating anything inside the quotes. Whatever Syntax we land on to make that happen I am open to. I am more focused on the concept..

With these examples it adds a lot of Flexibility where I don't need to "Invoke-NativeShell" as it would just return a string. Which means we are free to manipulate it just like a string would be. So that this potentially could become a Staple of what great looks like.

A Pivot like this also would mean we wouldn't have to deal with issues above where we spin our wheels on an almost solution when we can handles an edge case, vs addresses the problem as a whole.

Quick Examples:

# For this scenario let @!"<text>"!@ be our HereString Verbatim

Invoke-NativeShell @!"complicated native command with "quotes" and | pipe"!@
# or
Invoke-NativeShell @!"
Complicated native command with "quotes" and pipe and <tabs>
"!@

#or 
$r = @!"
complicated native command with custom arguments: {0}
"!@ -format "foo"
Invoke-NativeShell $r

That doesn't address the issue @mklement0 cover in the second part of this comment. Fact of the matter is, we can't just pass a single string to every comand ever; in many cases they have to be split up.

I believe if we add flexibility to Strings to handle "almost raw" unparsed data we wouldn't have issues about this pop up.

This is what ' and @' are for.

With these examples it adds a lot of Flexibility where I don't need to "Invoke-NativeShell" as it would just return a string. Which means we are free to manipulate it just like a string would be. So that this potentially could become a Staple of what great looks like.

Native commands return a sequence of strings. @!" has the potential of becoming a Staple of what unintelligible looks like.

I believe if we add flexibility to Strings to handle "almost raw" unparsed data we wouldn't have issues about this pop up.

This is what ' and @' are for.

With these examples it adds a lot of Flexibility where I don't need to "Invoke-NativeShell" as it would just return a string. Which means we are free to manipulate it just like a string would be. So that this potentially could become a Staple of what great looks like.

Native commands return a sequence of strings. @!" has the potential of becoming a Staple of what unintelligible looks like.

I think you are confused. With the point I was making.
Single quotes and Double quotes do only expand $variable or not. It does not automatically escape characters. It also limits me so I can no longer use its defining character. So instead For it to work properly as it is now. I would still have to escape characters which is the entire point of why we needed to include a &! Command.

If you have a better solution I am all ears. I do believe this hits more edge cases than adding a &!

Single quotes and Double quotes do only expand $variable or not. It does not automatically escape characters. It also limits me so I can no longer use its defining character. So instead For it to work properly as it is now. I would still have to escape characters which is the entire point of why we needed to include a &! Command.

Which characters would you have to escape and why?

@romero126 Now I'm confused what you are saying. Are you aware, that powershell has a here string?

I previously thought, that you might consider the syntax of powershells here strings annoying (At least I am sometimes annoyed, that there has to be a newline at the beginning and the end...), but now I'm not so sure what you are saying, could you clarify?

@TSlivede, agreed on here-strings being syntactically somewhat cumbersome.

There is an existing proposal, #2337, that proposes improvements, and while it is focused on allowing the closing delimiter to be indented, @lzybkr also proposes a Rust-like _single-line_(-also) variation in https://github.com/PowerShell/PowerShell/issues/2337#issuecomment-391152107, which would allows us to do the following, for instance:

# `ins` is `Invoke-NativeShell`
ins @' python -c 'print("hi")' | cat -n '@

The need for escaping would be avoided by combining _multiple_ ' with @, as needed (e.g., @'' can use @' here ''@.

In other words: this would be a general enhancement to PowerShell string literals, usable everywhere, which Invoke-NativeShell would benefit from as well.

Given that #2337 is primarily focused on something else, I've created a focused proposal in #13204 , and I suggest we continue the conversation there.

The need for escaping would be avoided

I've seen this a couple of times in this thread.

Any indicator to the tokeniser to begin a new mode would need an escape mechanism, since you need to differentiate the indicator to the tokeniser that you're leaving that mode from the literal form of that character.

The most natural form of that today is single-quoted strings:

• ': start single-quote tokenisation
• ': end single-quote tokenisation
• '': write a literal single quote within a single-quoted string

I think the reason we're talking about escaping despite this is:

• The --% operator defines too many ways to exit its mode (|, &&, ||, ;, newline) and allows strings within it (stripping their ' characters)
• Herestrings and co resolve to one string argument rather than an array of them

Wanting to pass arguments to an executable invocation mean we're actually talking about an array syntax:

• How to start the array (start verbatim mode)
• How to separate the array (pass a single argument to the new executable)
• How the end the array (end verbatim mode)

(This is a native concept on *nix, since exec expects an array. On Windows, I think we're forced to take an array, serialise it to a particular string syntax and pass it through CommandLineToArgvW. However, .NET provides an array-based abstraction for both of these -- so arrays make the most sense still)

That means we need two escapes:

• The array separator literal
• The array end literal

For example, let's say:

• --% before the command becomes the new verbatim syntax
• How do we end the syntax? newline and ; maybe?
• How do we separate arguments? maybe?

The questions then are:

• How do we pass a literal ; or newline?
• How do we pass a literal ?

A native command line is not an array. It is either a string or an AST. Since PowerShell does not know anything about ASTs in native shells, only a string remains.

A native command line is not an array. It is either a string or an AST

You haven't provided any grounds for this assertion.

Here's where PowerShell builds the AST for a command (of any kind, since it does not know until runtime when the command resolves what kind of command it is):

https://github.com/PowerShell/PowerShell/blob/f82f74d89811e91613a7450e536d652c5d8f784e/src/System.Management.Automation/engine/parser/Parser.cs#L6391-L6634

Notice that command elements (including parameters and arguments) are added into an array.

You can see this in action in PowerShell like this:

> { ping 192.168.0.1 }.Ast.EndBlock.Statements[0].PipelineElements[0].CommandElements

StringConstantType : BareWord
Value              : ping
StaticType         : System.String
Extent             : ping
Parent             : ping 192.168.0.1

StringConstantType : BareWord
Value              : 192.168.0.1
StaticType         : System.String
Extent             : 192.168.0.1
Parent             : ping 192.168.0.1

That array is compiled into expressions here:

https://github.com/PowerShell/PowerShell/blob/f82f74d89811e91613a7450e536d652c5d8f784e/src/System.Management.Automation/engine/parser/Compiler.cs#L4127-L4160

And then that is passed to a pipeline invocation here (notice Expression.NewArrayInit(typeof(CommandParameterInternal[]), pipelineExprs) is where the arguments are passed):

https://github.com/PowerShell/PowerShell/blob/f82f74d89811e91613a7450e536d652c5d8f784e/src/System.Management.Automation/engine/parser/Compiler.cs#L3798-L3805

At runtime, when the compiled expressions are run, that becomes this method call, which passes through to this call which decides what command processor to use:

https://github.com/PowerShell/PowerShell/blob/f82f74d89811e91613a7450e536d652c5d8f784e/src/System.Management.Automation/engine/runtime/Operations/MiscOps.cs#L29-L312

Eventually, when the pipeline is called, that argument array is bound to the native command using the NativeCommandParameterBinder:

https://github.com/PowerShell/PowerShell/blob/f82f74d89811e91613a7450e536d652c5d8f784e/src/System.Management.Automation/engine/NativeCommandParameterBinder.cs#L69-L137

That takes the argument array, which includes AST metadata like what kind of string was used for each argument, and builds a new string for argument invocation in the NativeCommandProcessor here:

https://github.com/PowerShell/PowerShell/blob/f82f74d89811e91613a7450e536d652c5d8f784e/src/System.Management.Automation/engine/NativeCommandProcessor.cs#L1094-L1166

This is done because the .NET API we currently use for process invocation is the old "all arguments in a single string" version, rather than the new one in .NET Core that allows you to pass an array and lets .NET rebuild the invocation as needed in a platform specific way.

But us using that API is an implementation detail. For example, we could call fork/exec directly on *nix, like we do here:

https://github.com/PowerShell/PowerShell/blob/f82f74d89811e91613a7450e536d652c5d8f784e/src/powershell/Program.cs#L268-L330

Notice that exec here takes an array -- we did that because it's the simplest and most direct way of passing arguments to the syscall, which is ultimately the goal of a verbatim argument syntax.

@rjmholt:

_Functionally_, all needs are already covered by the existing string-literal forms, _if you pass the native command line as a single string_, which I strongly recommend, via a new Invoke-NativeShell / ins cmdlet.

The starting point here is as _single string_, not an array: it is an _entire command line_ - to be parsed _by the target shell_ - and it makes the most sense to reflect this in PowerShell as such - as opposed to trying to awkwardly shoehorn a different shell's syntax into PowerShell's with something like --%

All that Invoke-NativeShell has to do is to pass the single string that contains the entire native-shell command line to the native shell via its CLI:

• on Unix, this means: pass the string as-is, _as whole_, as the second element of the argument _array_ passed to exec (with -c being the first argument-array element, and /bin/sh the executable); expressed in .NET terms with a simple example, where printf "'%s'" foo | cat -n is the verbatim content of the single string containing the command line to pass through (note the use of .ArgumentList, not .Arguments; the doubling of ' s an artifact of this example only):

  • $psi = [Diagnostics.ProcessStartInfo]::new('/bin/sh'); $psi.ArgumentList.Add('-c'); $psi.ArgumentList.Add('printf "''%s''" foo | cat -n'); [Diagnostics.Process]::start($psi).WaitForExit()

• on Windows, this means: to accommodate cmd's quirks, fill the lpCommandLine argument of CreateProcess as follows (with lpApplicationName assumed to be the value of environment variable ComSpec, which points to cmd.exe): "/c " + cmdLine, where cmdLine is the whole command-line string as-is; expressed in .NET terms with a simple example, where echo a^&b & ver is the verbatim content of the single string containing the command line to pass through:

  • $psi = [Diagnostics.ProcessStartInfo]::new($env:ComSpec); $psi.Arguments = '/c ' + 'echo a^&b & ver'; [Diagnostics.Process]::start($psi).WaitForExit()
    
    
    
    • @yecril71pl, this implies the answer to your question as to whether command-prompt or batch-file semantics are to be used: it is the former, as also used by other scripting languages, such as Python with os.system(); the (perhaps unfortunate) implication is that you won't be able to escape % characters, and that for loop variables must use a single % (e.g., %i instead of %%i); and that you must prefix loop-body commands with @ to prevent them from being echoed (e.g., for %i in (*.txt) do @echo %i).
    
    

This process will be _transparent to the end user_, so that all they need to focus on is to pass the native-shell command line _verbatim_, only needing to satisfy _PowerShell's_ string-literal syntax requirements:

With the form ins @'<newline>...<newline>'@, you already _can_ use your native-shell command lines as-is: see above for how the various existing string-literal forms can be used, which - due to the availability of the _interpolating_ here-string form (@"<newline>...<newline>"@) as well - includes the ability to use _PowerShell_ variables and expressions in the command line (the inability to do so is one of the major shortcomings of --%).

What my previous comment proposes is a _new general form of raw string literal_, which is _separate from this proposal_ (and its parsing logic is (hopefully) well-defined); though, as stated, Invoke-NativeShell calls would then be _more convenient_.

Instead of what you can already do (assuming an Invoke-NativeShell / ins command):

# ALREADY WORKS: Literal here-string
ins @'
python -c 'print("hi")' | cat -n
'@

with the proposed new single-line raw literal you could simplify to:

# PROPOSED, more convenient syntax: new raw literal.
ins @' python -c 'print("hi")' | cat -n '@

If the specifics of this new raw string-literal syntax need debating, please let us do it in #13204, not here.

So rereading the proposal to understand the original intent, I see it's really just asking for a better syntax for /bin/sh -c '<command>' + the cmd version, along with a special string escape logic for each target shell. I had based my initial conception on a special syntax and needing to work with the parser and native command processor, and on .NET's ProcessStartInfo.Arguments property being very difficult to use correctly in terms of quote passing.

But in fact the Arguments string just gets passed through verbatim. So all we need to do is pass such a string through directly.

In that case, I think any new syntax is totally unneeded (it will just complicate tokenisation logic and I suspect confuse people) and instead this is just a new command, ins like you say. .NET has some logic for this here which we'll be forced to reimplement, but that's ok since we need to be a little bit cleverer anyway.

Some comments on @rjmholt's comment https://github.com/PowerShell/PowerShell/issues/13068#issuecomment-660231641

Any indicator to the tokeniser to begin a new mode would need an escape mechanism, since you need to differentiate the indicator to the tokeniser that you're leaving that mode from the literal form of that character.

I agree and I don't think an in-line here string syntax can completely avoid the need to escape.

The --% operator defines too many ways to exit its mode (|, &&, ||, ;, newline) and allows strings within it (stripping their ' characters)

When speaking of --%, are you referring to the existing --% switch? If so, it's better to make that clear in your comment, as now --% is proposed to be a call operator.

Wanting to pass arguments to an executable invocation mean we're actually talking about an array syntax:

The proposal is to call the native shell (sh or cmd) and pass everything literally after the call operator --%, like sh -c .... So it's not powershell that passes the arguments to the target executable. In that case, I think we don't need _an array syntax_, right?

How do we end the syntax? newline and ; maybe?
How do we separate arguments? 'space' maybe?

I think the verbatim parsing ends at newline. To pass in a new line as part of the argument, use \n I guess, just like what you do in bash directly.

A native command line is not an array. It is either a string or an AST

You haven't provided any grounds for this assertion.

Here's where PowerShell builds the AST for a command (of any kind, since it does not know until runtime when the command resolves what kind of command it is):

If this proposal gets implemented, it will introduce a special construct where PowerShell will not build the AST at all.

Glad to hear it, @rjmholt.

.NET has some logic for this here which we'll be forced to reimplement

I don't think we actually need to reimplement anything, as the commands in my previous comment show: on Windows, use .Arguments for whole-command-line-string pass-through to the WinAPI; on Unix, use .ArgumentList to construct the argument _array_ that is used directly with exec.

If this proposal gets implemented, it will introduce a special construct where PowerShell will not build the AST at all.

Well it would build a different AST type, one designed for this purpose. If it's not part of the AST then it's basically the same as a comment, nothing happens.

_Functionally_, all needs are already covered by the existing string-literal forms, _if you pass the native command line as a single string_, which I strongly recommend, via a new Invoke-NativeShell / ins cmdlet.

I personally like the Invoke-NativeShell cmdlet idea, in some ways better than the call operator --%. The main concern I heard about this idea is that it's more typing than --%+Ctrl+v and differentiating literal here-string from interpolating here-string is another burden to user (_I personally think ins is way easier to type than --%_).

The main concern I heard about this idea is that it's more typing than --%+Ctrl+v and differentiating literal here-string from interpolating here-string is another burden to user (_I personally think ins is way easier to type than --%_).

Maybe PSRL can help. Similar to the issue about a key handler to surround pasted paths, PSRL could read the CommandAst, see it's ins and properly escape the pasted string.

Well it would build a different AST type, one designed for this purpose. If it's not part of the AST then it's basically the same as a comment, nothing happens.

Sure but the tree will be trivial. It will not reflect the meaning of the commands that are to be executed.

Start process cmd.exe redirect stdin / stdio and we can literally use a string to invoke-NativeCommand same with bash

Glad to hear it, @daxian-dbw.
Love the idea, @SeeminglyScience .

differentiating literal here-string from interpolating here-string is another burden to user

I think that's a _plus_, because - unlike --% - use of @"<newline>...<newline>"@ allows you to directly incorporate the values of _PowerShell_ variables and expressions into the native command line.

Of course, that assumes that users know the difference between single-quoted (verbatim) and double-quoted (interpolating) strings and about selective `-escaping of $, but I think it's an important (advanced) option to have.

To help the cut-and-paste users, the proposed PSReadLine feature should default to @'<newline>...<newline>'@ (verbatim).

Ultimately, we don't want to promote a mindset where users don't need to know and understand PowerShell's unique syntax requirements - instead, we want to promote understanding of them, in light of the _added power they bring_.

But to encourage users to use PowerShell's _own_ syntax, which is by definition _cross-platform_, it must work predictably.
To that end, #1995 must be fixed, ideally directly, allowing a breaking change, or, if that's not an option, via a minimal-ceremony _opt-in_.

To summarize:

• Something like --% (the new operator form proposed here), by definition a _platform-specific_ feature, is in no way a substitute for fixing #1995.

• Invoke-NativeShell (ins) avoids all the limitations of the proposed --% operator and provides the "trap door" solution that @joeyaiello mentioned: a quick way to execute a command line written for the native shell, without having to adapt it to PowerShell's syntax.

• --% should remain in its original _argument_ form only, and should remain a _Windows-only_ feature (which it _effectively_, albeit not _technically_ is) - in other words: no change needed.

  • Once #1995 is fixed, --%'s sole purpose will be to accommodate _edge cases on Windows_: allowing you to control the explicit quoting of the command line passed to rogue programs such as msiexec that require very specific forms of intra-argument quoting.

One of the other features I really want in PS v7.x is the ability to have a native command exiting with an error code stop my script, like set -e as discussed in this RFC. If the solution to that is something like an Invoke-NativeCommand, would that get confusing with Invoke-NativeShell?

Just trying to play the movie forward a bit because as much as I want the call native feature, I really want my build & test scripts to error out when calling msbuild, cl, cmake, Conan, npm, etc and those native commands exit with a non-zero exit code without having to remember to check $LASTEXITCODE all the time. If the implementation is via just another preference variable e.g. $PSNativeCommandInErrorActionPreference then I suppose that would impact the invocation of the native shell - it being a native "command" and all?

The main concern I heard about this idea is that it's more typing than --%+Ctrl+v and differentiating literal here-string from interpolating here-string is another burden to user (_I personally think ins is way easier to type than --%_).

Maybe PSRL can help. Similar to the issue about a key handler to surround pasted paths, PSRL could read the CommandAst, see it's ins and properly escape the pasted string.

But what about:

$c = gcm ins
& $c ...

PSRL is going to have a hard time recognizing what's going on there. It would have to be stateful, resolve the alias ins to invoke-nativeshell, then figure out that you want to call the commandinfo, then demunge the args? I really, really dislike the idea of special casing a fake command.

differentiating literal here-string from interpolating here-string is another burden to user

I think that's a _plus_, because - unlike --% - use of @"<newline>...<newline>"@ allows you to directly incorporate the values of _PowerShell_ variables and expressions into the native command line.

I'm not following you @mklement0. Why can't --% be taught to do this? I already gave this as a use case.

I really, really dislike the idea of special casing a fake command.

There is no fake command here, only a bona fide cmdlet, Invoke-NativeShell, aliased to ins.

gcm ins (Get-Command ins) will provide information about the ins alias in the form of a [System.Management.Automation.AliasInfo] instance, which you can later pass to & for invocation, as with any other command or alias.

Therefore, your example will work exactly as intended - it's just that if you use the gcm (Get-Command) detour, you'll have to spell out the appropriate string-literal syntax _yourself_, without the _optional convenience command scaffolding_ that PSRL _could_ provide, if @SeeminglyScience's suggestion is implemented:

$c = gcm ins
& $c @'
python -c 'print("hi")' | cat -n
'@

Or, if the new raw string-literal syntax proposed in #13204 is implemented:

$c = gcm ins
& $c @' python -c 'print("hi")' | cat -n '@

Of course, if you're familiar with the syntax of PowerShell's string literals, a regular single-quoted string will do too: simply double the embedded ' inside the overall '...' string:

$c = gcm ins
& $c 'python -c ''print("hi")'' | cat -n'

Why can't --% be taught to do this?

• --% is lacking a closing delimiter, which prevents it use in Powershell pipelines, which is an unacceptable restriction (note that (...) enclosure is _not_ an option, because that implies up-front collection of all output lines in memory rather than _streaming_ processing).

• On Unix, --% cannot _both_ support (unescaped) $ as a PowerShell metacharacter _and_ as a POSIX-shell metacharacter.

Again: there is _no_ justification for and no benefit to shoehorning a _different shell's syntax_ into PowerShell - pass the native-shell command line _as a string_, to ins, and all conceptual problems go away.

Invoke-NativeShell implies that PowerShell isn’t the native shell. Is that assumption always correct and always shall be?

@oising the use case is someone copy and pasting a native command from a README or something. If they're doing something more complicated than typing ins SPACE CTRL + v then they'll just build and escape their own string like normal.

@mklement0

But to encourage users to use PowerShell's _own_ syntax, which is by definition _cross-platform_, it must work predictably.
To that end, #1995 must be fixed, ideally directly, allowing a breaking change, or, if that's not an option, via a minimal-ceremony _opt-in_.

This is separate from the discussion on ins/--% right? If you disagree can you clarify why that issue would affect the new command/syntax?

@SeeminglyScience, yes, it is separate, but that hasn't been clear in this thread.
The only reason #1995 ever came up here is that it was originally closed in favor of this issue (though fortunately reopened since), and the OP here still states that this issue will fix it ("This should also solve these issues:").
Therefore, I wanted to stress that ins / --% would _not_ address #1995, which requires its own fix, urgently.
(Only if PowerShell's own argument passing works properly can we in good conscience recommend that users fully learn and embrace PowerShell's own syntax - which we should.)

@SP3269, we can cross that bridge when we get to it 😁.

But seriously:

If you think of "native" as "written specifically for the host platform [family]", the name would still fit, even if PowerShell should ever become the default system shell on a given platform (here's hoping, but I don't think that's realistic, at least not in the foreseeable future).

As for alternatives:

• Invoke-[System]DefaultShell would be the most accurate name (though it would unambiguously no longer apply should PowerShell ever become a system's default shell), but "native" seems to be the more commonly used term in the PowerShell world.

• Invoke-LegacyShell has some justification on Windows, but I don't think Unix users would take too kindly to that term.

I think I am beginning to unravel the utter confusion I'm experiencing trying to keep track of everyone's motives, ideas and issues here. It seems that we're (@mklement0 and I, at least) are looking at this as two different solutions to two different views of the same conceptual problem. What I was suggesting was using --% as a hint to the _parser_ to change the way things are parsed in conjunction with the call & operator (not necessarily using --% standalone.) On the other hand, Michael seems to be looking at ins as a drop-in cmdlet to replace powershell's own internal native command broker and argument parser, and is not looking to change powershell's parser, instead, using strings/here-strings to capture the intended parameters (which I am saying could be used with --% also.)

--% is lacking a closing delimiter, which prevents it use in Powershell pipelines

This point I don't get at all. It's no more lacking a closing delimiter than ins/invoke-nativecommand is. If you need to delimit or feed from the LHS, use here-strings, else it is considered as a single statement.

Regardless of whatever way is chosen, it seems that there will be a need to choose a native shell to dispatch the commandline. I suggest we use the environment variables COMSPEC on Windows (defaults to %systemroot%\system32\cmd.exe) and SHELL on Linux (defaults to /bin/bash) -- if SHELL doesn't exist, we should target /bin/sh (which in most cases is a symlink to bash anyway.)

--% is lacking a closing delimiter, which prevents it use in Powershell pipelines

This point I don't get at all. It's no more lacking a closing delimiter than ins/invoke-nativecommand is. If you need to delimit or feed from the LHS, use here-strings, else it is considered as a single statement.

Basically you can't call things like cmd --% /c someapp | someOtherApp and have cmd handle the pipeline; PowerShell still interprets a pipeline (and a few other things like && and || which would otherwise be useful for Unix folks) as being a PowerShell token instead of passing it to the native command as part of the argument string.

and SHELL on Linux (defaults to /bin/bash)

No: the _system_ shell on Unix-like platforms is _invariably_ /bin/sh.
This is distinct from a given _user's_ interactive shell (reflected in $env:SHELL - but, unfortunately, currently not if _PowerShell_ is the user's shell, see #12150), which is (a) configurable and (b) often _defaults to_ /bin/bash, but even that is not a given, as evidenced by macOS recently transitioning to /bin/zsh as the default interactive shell for new users.

I suggest we use the environment variables COMSPEC on Windows

Good point, that's the better way to refer to the system shell (command interpreter) on Windows - I've updated the sample command above accordingly.

One of the other features I really want in PS v7.x is the ability to have a native command exiting with an error code stop my script, like set -e as discussed in this RFC. If the solution to that is something like an Invoke-NativeCommand, would that get confusing with Invoke-NativeShell?

Just trying to play the movie forward a bit because as much as I want the call native feature, I really want my build & test scripts to error out when calling msbuild, cl, cmake, Conan, npm, etc and those native commands exit with a non-zero exit code without having to remember to check $LASTEXITCODE all the time. If the implementation is via just another preference variable e.g. $PSNativeCommandInErrorActionPreference then I suppose that would impact the invocation of the native shell - it being a native "command" and all?

We have Start-NativeExecution in our build module for that purpose.

I've seen that. How exactly would you combine Start-NativeExecution with Invoke-NativeShell if you'd like the later to effectively throw on a non-zero exit code? I really hope that something like PR #3523 makes it in along with this call native feature because I want the two to work together. I suppose if call native winds up being implemented as a cmdlet (Invoke-NativeShell vs --%) then -ErrorAction Stop could be implemented to have it throw (terminating error) on a non-zero exit code.

--% is lacking a closing delimiter, which prevents it use in Powershell pipelines

This point I don't get at all. It's no more lacking a closing delimiter than ins/invoke-nativecommand is. If you need to delimit or feed from the LHS, use here-strings, else it is considered as a single statement.

Basically you can't call things like cmd --% /c someapp | someOtherApp and have cmd handle the pipeline; PowerShell still interprets a pipeline (and a few other things like && and || which would otherwise be useful for Unix folks) as being a PowerShell token instead of passing it to the native command as part of the argument string.

Yes, I understand the current behaviour @vexx32 . But we're not (at least I'm not) talking about the current behaviour of --% -- we're talking about enhancing it, no? Why do I get the feeling that we're all talking around each other here? :)

As I said way, way up above, we could enhance & to work with --% so this is possible. I also think we need to be clear about implicit native exec versus explicit invocation of a shell with arguments. This is a constant source of confusion - even for seasoned windows users - that somehow cmd.exe (a shell) is needed to "run" other executables; the same applies to linux/osx.

Yes, I understand the _current_ behaviour @vexx32 . But we're not (at least I'm not) talking about the _current_ behaviour of --% -- we're talking about enhancing it, no? Why do I get the feeling that we're all talking around each other here? :)

So the current pitch for adjusting --% is that if it's where the command name would typically be, everything to the right of it is parsed as is (up to a new line) and sent directly to bash/cmd. There's no room any PowerShell syntax like here-strings and what not because then it has all the same problems that --% and the native command processor currently have.

I've seen that. How exactly would you combine Start-NativeExecution with Invoke-NativeShell if you'd like the later to effectively throw on a non-zero exit code? I really hope that something like PR #3523 makes it in along with this call native feature because I want the two to work together. I suppose if call native winds up being implemented as a cmdlet (Invoke-NativeShell vs --%) then -ErrorAction Stop could be implemented to have it throw (terminating error) on a non-zero exit code.

Currently it will be Start-NativeExecution { Invoke-NativeShell @whatever }.

This point I don't get at all. It's no more lacking a closing delimiter than ins/invoke-nativecommand is. If you need to delimit or feed from the LHS, use here-strings, else it is considered as a single statement.

Invoke-NativeShell does not _need_ any closing delimiter because it is a regular command, whereas the horror of --% is not.

and SHELL on Linux (defaults to /bin/bash)

No: the _system_ shell on Unix-like platforms is _invariably_ /bin/sh.

I think the thing should be equivalent to making an executable script file and running it, which delegates the task of choosing the right shell to the operating system and we should not be bothered. Including, if it starts with #!, so be it.

The idea behind Invoke-NativeShell is to provide a convenience wrapper around the native shell's _CLI_.

• On Unix, that is fully sufficient, and creating an auxiliary script file not only introduces extra overhead, but then reports a random value in $0 (the script file path), whereas invocation via the CLI it predictably contains /bin/sh and can even be set explicitly by following the code-argument with another argument (e.g., sh -c 'echo $0' foo)

  • (As an aside: creating a shell script with shebang line #!/bin/sh means targeting the system shell by full path just as explicitly as invoking /bin/sh directly).

• On Windows, the execution via a batch file _would_ bring advantages (but not for delegating the task of locating the system shell, which $env:ComSpec predictably does), as it would avoid the problems with % escaping and for behavior mentioned above.

To address the latter as a courtesy (so that users don't have to write their own aux. batch files), for conceptual clarity, we could offer an
-AsBatchFile switch as an _opt-in_.

That said, if there is consensus that the majority of public cmd command lines used for cut-and-paste are written with _batch-file_ semantics in mind (%%i rather than %i as a loop variable, ability to escape a verbatim % as %%), perhaps invariably using an aux. batch file _on Windows_ (while sticking with the CLI on Unix) is the better solution - I do not feel strongly about this, but it would have to be clearly documented, especially given that it means exhibiting behavior that is different from other scripting languages.

@rkeithhill

To me, the integration-of-native-errors RFC you link to is worth addressing just as urgently as #1995:

Both represent necessary steps toward making external programs (native utilities) first-class citizens in PowerShell (as much as conceptually possible), _which they should always have been_.

Being a first-class citizen means: _direct_ invocation (with & needed only for _syntactic_ reasons, situationally), not _via a cmdlet_.
In other words: there should never be an Invoke-NativeCommand cmdlet or a Start-NativeExecution cmdlet.

(As an aside: Start-NativeExecution is misnamed, as many functions in the build module are; Start signals _asynchronous_ operation, whereas most of these function are _synchronous_ - see the discussion about Start-Sleep at https://github.com/MicrosoftDocs/PowerShell-Docs/issues/4474).

Therefore, Invoke-NativeShell requires no special consideration: PowerShell, as it already does, will set $LASTEXITCODE based on the exit code reported by the native-shell executable process (cmd / sh) invoked.

Then the mechanism proposed in the RFC will act on it, as it would act on directly invoked executables (e.g., if $PSNativeCommandErrorAction = 'Stop' is set, a script-terminating error would occur if $LASTEXITCODE is nonzero.)

(A quick aside - this discussion doesn't belong here: As for a _per-call_ mechanism: something like
/bin/ls nosuch || $(throw 'ls failed) works, as does /bin/ls nosuch || $(exit $LASTEXITCODE) (the equivalent to a POSIX shell's /bin/ls nosuch || exit); #10967 discusses why we unfortunately cannot avoid $(...) (without major changes to the grammar); similarly, needing to refer to $LASTEXITCODE explicitly probably cannot be avoided.)

(As an aside: creating a shell script with shebang line #!/bin/sh means targeting the system shell by full path just as explicitly as invoking /bin/sh directly).

That is not what I wanted to say. I wanted to say that PowerShell does not need to know what the default shell is because operating systems have this feature built in. Linux knows how to execute an executable script in case it does not start with #! or it starts with something else, like #!/usr/bin/env python and Windows knows what to do to call a .CMD script, so we should not peek into%COMSPEC%’n’stuff either IMHO. Windows can also execute other scripts but it is based on the script file’s extension, so it obviously does not apply to this case.

Again, Invoke-NativeShell's mandate should be calling the native shell's _CLI_, not creating auxiliary script files (with side effects) behind the scenes (except for different reasons, as discussed above).

With respect to robustly determining the system shell, there is no problem to solve here: hard-coding /bin/sh on Unix is perfectly appropriate, as is consulting $env:ComSpec on Windows.

No, Unix platforms at the system-call level do _not_ know how to execute an executable plain-text file _without a shebang line_; that you still _can_ invoke such files is a convenience feature of POSIX-like shells: they try exec, then _fall back_ to interpreting such files as written _for them_; that is, it is whatever POSIX-like shell that is being run that ends up executing the file - whereas PowerShell simply _fails_, because it doesn't implement this courtesy fallback (you'll get Program 'foo' failed to run: Exec format error).

Needless to say, it is therefore ill-advised to create such scripts.

No, Unix platforms at the system-call level do _not_ know how to execute an executable plain-text file _without a shebang line_; that you still _can_ invoke such files is a convenience feature of POSIX-like shells: they try exec, then _fall back_ to interpreting such files as written _for them_; that is, it is whatever POSIX-like shell that is being run that ends up executing the file - whereas PowerShell simply _fails_, because it doesn't implement this courtesy fallback (you'll get Program 'foo' failed to run: Exec format error).

csh is not POSIX-like and it does not fail on exec a bare script. It invokes sh. So does perl.

I hope it is obvious that this makes my point: it's _up to each shell_ to decide what to do, and different shells / scripting languages do different things; PowerShell currently just fails - if it had to make a choice, and you wanted that choice to be /bin/sh, we're back at square one: /bin/sh must be assumed.

I hope it is obvious that this makes my point: it's _up to each shell_ to decide what to do, and different shells / scripting languages do different things; PowerShell currently just fails - if it had to make a choice, and you wanted that choice to be /bin/sh, we're back at square one: /bin/sh must be assumed.

exec in perl is a direct system call, perl does not modify it in any way. In particular, it does not _choose_ the interpreter.

Of course, that assumes that users know the difference between single-quoted (verbatim) and double-quoted (interpolating) strings and about selective `-escaping of $, but I think it's an important (advanced) option to have.

Maybe it is just me, but I would trade a thousand "’s for one -f.

@yecril71pl

exec in perl is a direct system call, perl does not modify it in any way. In particular, it does not choose the interpreter.

That can not be true if it works with scripts without #! line. The exec syscall on linux does not work without #!, as one can easily test:

user@Programming-PC:/tmp/exec_test$ ls
test.c  testscript
user@Programming-PC:/tmp/exec_test$ cat test.c
#include <unistd.h>
#include <stdio.h>

int main(int argc, char **argv) {
  char *binaryPath = "./testscript";

  execl(binaryPath, binaryPath, NULL);

  perror("Error");

  return 0;
}
user@Programming-PC:/tmp/exec_test$ gcc test.c -o testexecutable
user@Programming-PC:/tmp/exec_test$ chmod +x testscript 
user@Programming-PC:/tmp/exec_test$ cat testscript 
echo test from script
user@Programming-PC:/tmp/exec_test$ #works from shell:
user@Programming-PC:/tmp/exec_test$ ./testscript 
test from script
user@Programming-PC:/tmp/exec_test$ #doesn't work via exec syscall:
user@Programming-PC:/tmp/exec_test$ ./testexecutable 
Error: Exec format error
user@Programming-PC:/tmp/exec_test$ vim testscript 
user@Programming-PC:/tmp/exec_test$ cat testscript 
#!/bin/sh
echo test from script
user@Programming-PC:/tmp/exec_test$ #exec syscall works with #! line:
user@Programming-PC:/tmp/exec_test$ ./testexecutable 
test from script
user@Programming-PC:/tmp/exec_test$ uname -a
Linux Programming-PC 4.4.0-176-generic #206-Ubuntu SMP Fri Feb 28 05:02:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
user@Programming-PC:/tmp/exec_test$ 

Edit: I just noticed, that if you use the library function execlp then according to the man page:

If the header of a file isn't recognized (the attempted execve(2) failed with the error ENOEXEC), these functions will execute the shell (/bin/sh) with the path of the file as its first argument. (If this attempt fails, no further searching is done.)

However this is not a feature of "linux" but of the gnu c library - and again according to the man page it uses explicitly /bin/sh and not some userdefinable shell. (Hardcoded in the source of posix/execvpe.c (new_argv[0] = (char *) _PATH_BSHELL;) and sysdeps/unix/sysv/linux/paths.h (or sysdeps/generic/paths.h - I don't know which header is used...) (#define _PATH_BSHELL "/bin/sh"))

However this is not a feature of "linux" but of the gnu c library - and again according to the man page it uses explicitly /bin/sh and not some userdefinable shell. (Hardcoded in the source of posix/execvpe.c (new_argv[0] = (char *) _PATH_BSHELL;) and sysdeps/unix/sysv/linux/paths.h (or sysdeps/generic/paths.h - I don't know which header is used...) (#define _PATH_BSHELL "/bin/sh"))

Since it comes from sysdeps/**/linux, it is a Linux thing. It may also be the generic thing but it is overridable — obviously not locally but per OS.

Yes, I understand the _current_ behaviour @vexx32 . But we're not (at least I'm not) talking about the _current_ behaviour of --% -- we're talking about enhancing it, no? Why do I get the feeling that we're all talking around each other here? :)

So the current pitch for adjusting --% is that if it's where the command name would typically be, everything to the right of it is parsed _as is_ (up to a new line) and sent directly to bash/cmd. There's no room any PowerShell syntax like here-strings and what not because then it has all the same problems that --% and the native command processor currently have.

Oh lordy... Is everyone just trying to troll me? 😄 No, Patrick -- if you want to use powershell expressions, then use here-strings with --%. If you want multiline, use here-strings. If you want multline without variable substitution, you could use single-quote here-strings.

Anyway, I don't have any skin in this game and I think I've said enough. I still think there's something seriously wrong about using a cmdlet to defer to another shell from within a shell. It's just clumsy. It's especially clumsy when you're using a cmdlet to execute another shell to execute a native command that has no need for that secondary shell in the first place. The whole thing just smells bad.

edit: last word

The title of this item is "call native operator." I see a wasted opportunity when we already have a call operator & and we already have a token that was designed to deal with passing arguments to native commands, --%. Right now, using & with --% is NOT recognized and as such will NOT introduce a breaking change should we decide to enable this scenario and give it specific behavior to fix the problems being discussed here.

if you want to use powershell expressions, then use here-strings with --%. If you want multiline, use here-strings.

The point of the operator is that it doesn't parse literally any PowerShell syntax. There's no such thing as here-strings. There's no such thing as single quoted string constants. It's "send the text to the right as is" not "evaluate the expression to the right, convert to string and send it's value"

So for example, if you had this:

--% @'
echo my native command
'@

it would translate to:

cmd.exe /c "@'"
echo my native command
'@

Except you'd get a parser error because '@ would just be that start of a single quoted string constant containing @.

I think maybe you're picturing the implementation differently so that it's more or less the same thing as the pitch for ins, but the above is the current pitch being discussed for --%.

Also worth noting that this is how the existing functionality works as well. This example works mostly the same way:

cmd /c --% @'
echo my native command
'@

I repeat:

Right now, using & with --% is NOT recognized/undefined and as such will NOT introduce a breaking change should we decide to enable this scenario and give it specific behavior to fix the problems being discussed here.

Let me spell it out even clearer: Allow here-strings to follow --% when used with &

Hah that's a big oof - yeah I missed that, sorry @oising.

About that idea I think it would be a little confusing to have --% mean stop parsing in some contexts and "do a better job at native argument binding" in others (likewise with making it work like Invoke-NativeShell if that's what you mean instead).

Thanks for digging deeper re exec, @TSlivede. Let me try to summarize and therefore hopefully close this tangent; applies to both Linux and macOS:

There is no system (or library) function called exec on Unix, there is only a _family_ of related functions with exec in their name (most of them as a prefix).

• The _system_ functions (man section 2), execve at its heart, _fail_ when you try to execute a shebang-less script.

• Among the _library_ functions that _build on the system functions_ (man section 3 - man 3 exec), only those that have p (for "path", I presume) have the pass-to-the-system-shell fallback (e.g., execlp).

  • The GNU library implementations of these functions hard-code /bin/sh, as @TSlivede has shown, whereas the BSD-based ones use just sh, and rely on sh to be in $env:PATH (though, curiously, the man page states /bin/sh).

As stated, for _direct execution_ the major POSIX-like shells (bash, dash, ksh, and zsh) fall back to executing shebang-less scripts _themselves_, which implies that they do _not_ use the fallback variants among the exec library functions; by contrast, perl's choice for its own exec function was to rely on the fallback; similarly, the exec _builtins_ of the major POSIX-like shells rely on the fallback, except in bash.

Since on Unix the use of a script file behind the scenes is unnecessary, we can make the same assumptions about the system shell path as the fallback library functions, and I recommend /bin/sh over sh, for security and predictability:

Despite the POSIX spec _not_ mandating the location of sh (that is, strictly speaking, the BSD library functions are the compliant ones):

• /bin/sh is safe to assume, because it is the _de facto_ standard location, not least because writing portable Unix shell _necessitates_ referring to sh by its full path, given that shebang lines support only _full, literal_ paths (#!/bin/sh).

• Conversely, relying on locating sh via $env:PATH can be a security risk, given that $env:PATH could be manipulated to invoke a different sh.

The same risk applies to locating cmd.exe via $env:ComSpec, by the way, so perhaps the better way is to call the GetSystemDirectory WinAPI function and append \cmd.exe to its result (we need a fallback anyway, if $env:ComSpec happens not to be defined).

/bin/sh is safe to assume, because it is the _de facto_ standard location, not least because writing portable Unix shell _necessitates_ referring to sh by its full path, given that shebang lines support only _full, literal_ paths (#!/bin/sh).

Normally you say #!/usr/bin/env sh (except in system scripts).

• Normally, you _don't_: googling "#!/bin/sh" yields about 3,900,000 matches, "#!/usr/bin/env sh" yields about 34,100 matches.

• Normally, you _shouldn't_: you want to predictably target _the_ system shell, /bin/sh, not whatever sh utility happens to come first in $env:PATH.

The only reason to target an executable named sh is to portably target the lowest-common-denominator-assume-POSIX-features-only _system_ shell, i.e. /bin/sh.

Normally, you _don't_: googling "#!/bin/sh" yields about 3,900,000 matches, "#!/usr/bin/env sh" yields about 34,100 matches.

There is no way to limit a Web search to user scripts, especially since many user scripts are not marked executable at all, and even if they are, they may rely on execlp; but even if most user scripts said that, we should not take _customary_ for _normal_. The scripts to be run by PowerShell are user scripts; when users want a shell, they call sh, not /bin/sh, unless they are paranoid.

@oising

Passing the native-shell command line _as a single string_ (in whatever string(-literal) form is easiest), and _only_ as a single string (save for the additional, pass-to-the-command-line arguments supported on Unix (only) discussed above) - assuming you agree with that - sounds like common ground.

Passing the command line as a _single string_ is the prerequisite for:

• being able to incorporate such a call into a _PowerShell_ pipeline.
• being able to incorporate _PowerShell variables and expression_ into the native-shell command line.

Passing a single string is a _direct conceptual expression_ of what the call will do; by contrast, trying to somehow directly incorporate a different shell's syntax directly in PowerShell's with _individual_ (optionally bare-word) arguments brings unsolvable problems that result in a confusing compromise with fundamental limitations, which the existing --% already embodies.

While you could argue that it then isn't all that important whether we choose an _operator_ or a _cmdlet_ to pass that single-string command line to (save for questions of _discoverability_), I have conceptual concerns about use of both & and --% (preceded by &):

• The concern re &: if we need to pass a - quoted - string containing the command line, we're effectively operating in _expression_ mode, whereas the & currently implies _argument_ mode (which means: _individual_ arguments, quoting only when necessary). Therefore, the involvement of & is confusing.

• The concern re --% is about the Windows-only origin and muddled semantics of --% as the stop-parsing symbol; it would be especially confusing if --% as the stop-parsing symbol operated in _argument_ mode (e.g.
  & /bin/ls --% dir1 dir2), whereas & --% operated in _expression_ mode (e.g. & --% '/bin/ls dir1 dir2 >out.txt')

Let me take a step back and examine --%, the stop-parsing symbol, as currently implemented:

It was an attempt to solve two ultimately unrelated problems (Windows-only at the time):

• (a) A use case that also applies to other platforms: Let me reuse _command lines_ written for cmd.exe / the native shell as-is, so I don't have to adapt them to PowerShell's syntax.

• (b) An always Windows-only problem: Let me call a _rogue CLI_ (that is, a call to a _single_ external console application) that requires a very specific style of quoting that PowerShell's generic behind-the-scenes re-quoting cannot provide, by allowing me to craft the command line ultimately passed to the WinAPI for such CLIs _verbatim_. (To limit the problem statement to this assumes that PowerShell's re-quoting generally works properly, which it never has - that is the subject of #1995).

--% was implemented as a muddled _conflation_ of solution attempts to both (a) and (b), and ended up solving neither problem properly:

• It was advertised as the solution to (a), but in reality has severe limitations, because _the only cmd exe feature that it emulates is the expansion of %...%-style environment-variable references_:

  • It only ever supports a _single_ cmd command, given that an | (and
    && and ||, even though they weren't implemented at the time) implicitly terminate the pass-to-cmd portion.

  • A _single_ & - which is cmds multi-command separator - _is_ passed through, but that too did _not_ result in multi-command support, because - since cmd isn't actually involved - the & is passed _verbatim to the target program_; e.g.
    echoArgs.exe --% a & b doesn't echo a then invokes b, it passes verbatim arguments a, &, and b to echoArgs.

  • cmd's escape character (^) in unquoted arguments isn't honored.

  • As a corollary, %...% tokens that refer to existing environment variables are _invariably_ expanded; an attempt to prevent that via ^ doesn't work properly (e.g.
    echoArgs.exe Don't expand %^USERNAME%, which in cmd prevents expansion _and strips the ^_, retains the ^ when called from PowerShell as echoArgs.exe --% Don't expand %^USERNAME%

• As a solution to (b), it fell short as well:

  • Just like in the (a) use, there's no ability to incorporate _PowerShell_ variables and expressions into the command - except if you awkwardly first define an _aux. environment_ variable_ that you then reference via %...% after --%; e.g.
    $env:FOO='foo'; echoArgs.exe --% %FOO%!

The proper solution to (a) would have been Invoke-NativeShell / ins, as discussed here.

The proper solution to (b) would have been to:

• support --% as special only as the _first_ argument
• and, just like with ins, require it to be followed _by a single string_ that constitutes the pass-through command line _as a whole_, again with the ability to incorporate _PowerShell_ variables and expressions by way of string interpolation (or other ways of constructing the string)
• e.g. msiexec --% "/i installer.msi INSTALLLOCATION=`"$loc`"", with $loc containing 'c:\foo\bar none', would result in verbatim command line /i installer.msi INSTALLLOCATION="c:\foo\bar none" getting passed through, satisfying msiexec's nonstandard requirement that in <prop>=<value> arguments only the <value> part be double-quoted.

While that isn't _convenient_, it is the price to pay for a robust solution to the anarchy that is command-line-based argument passing on Windows.

@mklement0 Thank you for the patient and thoughtful reply (as usual.)

So, I wholeheartedly agree with;

The proper solution to (b) would have been to:

• support --% as special only as the first argument
• and, just like with ins, require it to be followed by a single string that constitutes the pass-through command line as a whole, > - again with the ability to incorporate PowerShell variables and expressions by way of string interpolation (or other ways of constructing the string)
  
  
  
  • e.g. msiexec --% "/i installer.msi INSTALLLOCATION="$loc"", with $loc containing 'c:foobar none', would result in verbatim command line /i installer.msi INSTALLLOCATION="c:foobar none" getting passed through, satisfying msiexec's nonstandard requirement that in = arguments only the part be double-quoted.
  
  

This is what I've been trying to expound as a generalized solution to both (a) and (b), except I still don't really get that (a) should exist as an independent problem to solve? What I'm really struggling with is if (b) is implemented _specifically_ for &, why can't that cover (a) also? e.g. What does ins ... give that & cmd --% ... cannot? And it won't be a breaking change other than not allowing & to pass --% as an argument (which seems ridiculously unlikely.) Bonus that you don't have to worry about comspec, shells or whatever. Let the caller decide.

@PowerShell/powershell-committee reviewed this:

• This will be an experimental feature to get real usage feedback; we are still open to the actual sigil; we defer /bin/sh vs /bin/env sh to the PR review
• We will move forward with --% as a new operator specifically for the "stackoverflow" scenario where the user cuts and pastes a command-line into PowerShell and it should just work (with, of course, the new operator preceding it)
• A Invoke-NativeShell cmdlet is separate from this proposal and could be published by the community in PowerShellGallery; also, requiring a here-string doesn't solve the user experience problem without knowing ahead of time to wrap it as a here-string
• We do not want to make a breaking change to --% switch where existing users may depend on that behavior
• Users who need streaming behavior like --% switch should continue to use that switch and escape special characters as necessary
• There is still a discovery problem for users to know about --% (or any solution)
• We are not currently entertaining a change to PSReadLine to modify pasted content as a here-string trying to predict user intent
• We are not looking at multiple solutions that allow for here-string vs non-here-string (expansion) scenarios; e.g. & --% vs --%

I also want to add that, despite @SteveL-MSFT and I closing #1995 at the same time as opening this one, it really wasn't our intent to communicate this as a definitive solution to that problem.

It's my view that the benefits of this particular feature are much more significant than the impact #1995. I think there are a LOT of StackOverflow examples and docs examples for non-PS enlightened tools that folks would like to cut/paste (myself included).

There's additionally a desire to provide proper escaping within PowerShell's own language, but I'm not seeing people hit #1995 in the wild en masse (but I'll elaborate more on this over there).

• We will move forward with --% as a new operator

Sorry to drag this out, and I'm not looking for additional discussion, but what kind of operator? (I'm primarily interested in syntax.)

$a = --% find . -iname *$pdf -print0 | xargs -0 ls -ltr

Who gets the pipe? What happens to the $? Similar with &, &&, and ||.

Under what conditions, if any, will --% be usable in a PS pipe moving forward?

Thanks.

Sorry to drag this out, and I'm not looking for additional discussion, but what kind of operator? (I'm primarily interested in syntax.)

AST wise it probably won't technically be an operator but it's own AST type. If it does get implemented as an operator, it'd be a unary operator.

$a = --% find . -iname *$pdf -print0 | xargs -0 ls -ltr

Who gets the pipe? What happens to the $? Similar with &, &&, and ||.

The string find . -iname *$pdf -print0 | xargs -0 ls -ltr would be sent as is to the native shell. $a would still be populated though, so you could in the next line pipe that to something.

Under what conditions, if any, will --% be usable in a PS pipe moving forward?

If ~in the first command element slot~ it's at the start of a statement, probably not. That's one of the draws of the new syntax. If not ~in the first slot~ at the start, it'll continue to work like it does today.

(@daxian-dbw @SteveL-MSFT if any of that is wrong please correct ❤️)

@SeeminglyScience is exactly right

If in the first command element slot, probably not. That's one of the draws of the new syntax. If not in the first slot, it'll continue to work like it does today.

It may be tricky to pass the output of upstream command to the native command that will be invoked by --%. --% basically just hands the string as it to a native shell, like bash, but the output from upstream command cannot just be fed to bash, but should be fed to the native command itself, such as find.

I think streaming behavior is not a consideration to this feature as it's mainly for the "StackOverflow" scenario. So I think we'd better just make --% a special statement ast, so that it's clear it won't work with _PowerShell_ pipelines.

Ahh I knew there was a reason that "command element slot" was wrong. I meant at the start of a statement. I'll fix it, thanks!

the output from upstream command cannot just be fed to bash

Does _upstream_ mean _us_? I think can do: 'ls' | bash. Not a peculiarly smart thing to do but possible.

Was discussing with @jpsnover about this and he proposed the shebang #! as the sigil. In this case, you would just type:

#!/bin/sh ls | grep foo
#!sh | grep foo

(where the second case assumes sh is in the path). In this case, we would have to decide if specifying the shell is required or if we always run that under the default shell, so in this example, sh is started twice. However, there is a problem with this due to how shebang files work today. They work in PowerShell because the shebang is ignored as a comment. If we decide to ignore the first line in a script if it's a comment, then we still have a problem in the interactive shell where each new set of lines is a new script and there's no way today to determine if that script is run as a script or interactively.

An alternative might be to borrow markdown syntax:

powershell ```bash ls | grep foo ```

In this example, we would use markdown code fencing syntax. This would also solve the multi-line problem and would use a concept that isn't entirely new. Given that we still have initial discovery problem for new users, I don't think this is that much worse in that you need to have the trailing triple-backticks.

In this case, this could potentially work:

powershell $a = ```bash ls | grep foo ``` | select-string bar

@SteveL-MSFT Markdown Syntax makes a lot of sense. If we are able to specify. other types as well such as python or perhaps by extension?

The shebang #! Would interfere with existing comments.

That seems massively overcomplicated to me, both to handle in parsing and for users to actually _use_.

And yeah, we don't need to confuse folks with lines that _should_ be comments not turning out as comments. #Requires is already edging on kind of weird, a full on shebang is going to confuse Windows folks at best, and probably Linux folks too.

I personally would prefer to not mixing this feature with shebang, it's confusing in my opinion.
Also, if we start to use the markdown code-block syntax, then it's just a matter of time for people to request running arbitrary language code directly in PowerShell. I don't think we want to go that way ...

Also, if we start to use the markdown code-block syntax, then it's just a matter of time for people to request running arbitrary language code directly in PowerShell.

I would literally never stop asking for inline C# even though I know it's a bad idea.

The markdown syntax does not introduce arbitrary language code because the embedded script will run in a child process. You can place arbitrary language code inside @' now and nothing bad happens.

In the team discussion and also with Jeffrey, we should not be against people wanting to run other languages within a PowerShell script. As @yecril71pl noted, we are not directly supporting other languages but relying on a different process to interpret and execute that code. The scenario here is that a user sees a block of python or bash script and just wants to use it as-is within their PowerShell script. There's nothing requiring people to do this. They can decide to port that other script to PowerShell if they prefer. This just provides an option for users to be able to cut and paste into PowerShell to get stuff done.

We don't need to mix the single-line vs multi-line proposals in that we could support both although the downside is now we have two ways of doing very similar things but with different syntax.

I would prefer to drop the single-line stuff. Alien code islands should be prominent, otherwise nobody will understand what is going on.

@yecril71pl I'm leaning towards that myself primarily because from a discovery standpoint, the markdown syntax I think would be more easily recognizable, but that might just be because I write markdown often. I also expect that lots of examples scripts are probably multi-line rather than just a single line as they expect some state to be retained where each single line would be an independent process.

In the team discussion and also with Jeffrey, we should not be against people wanting to run other languages within a PowerShell script.

Absolutely 100%, but that doesn't mean it needs to be built directly into the language.

As @yecril71pl noted, we are not directly supporting other languages but relying on a different process to interpret and execute that code. The scenario here is that a user sees a block of python or bash script and just wants to use it as-is within their PowerShell script.

That can already be done with here-strings right? Can you elaborate a little bit on what benefits this provides over existing methods?

Also why stop at python and bash? Why not C#, CIL, C++?

There's nothing requiring people to do this. They can decide to port that other script to PowerShell if they prefer. This just provides an option for users to be able to cut and paste into PowerShell to get stuff done.

I'm not against this because I think I'm going to be forced to use it. I don't like it because it's going to be difficult to maintain from a language perspective, a nightmare for editors, and ultimately encourage scripts that are unreadable unless you know several languages.

Really though this is a pretty different concept from the original topic of this thread with much wider implications. I'd recommend creating a new issue for it.

That can already be done with here-strings right? Can you elaborate a little bit on what benefits this provides over existing methods?

A universal code editor will be able to detect them and decorate them appropriately, whereas there is nothing to be done with @' because the editor has no idea what is inside.

Also why stop at python and bash? Why not C#, CIL, C++?

Because they are not scripting languages?

A universal code editor will be able to detect them and decorate them appropriately, whereas there is nothing to be done with @' because the editor has no idea what is inside.

An editor can tell that bash -c @' contains bash just as easily as a code fence. Also the challenging part isn't determining what language a section of code is, it's juggling language servers and syntax highlighters. The only way it would work without significant time and effort investment would be to essentially render it like it was a here-string.

Because they are not scripting languages?

C# and C++ both have script oriented subsets. Even if they didn't, "scripting language" is subjective with no real definition. It isn't helpful as a definitive boundary for what would and would not be considered for inclusion.

C# and C++ both have script oriented subsets. Even if they didn't, "scripting language" is subjective with no real definition.

A language is a (stand-alone) scripting language when you normally call interpet options… script arguments… and that call leaves nothing except what it was intended to leave, excluding temporary stuff and private to the interpreter. It also means that it normally requires a copy of the interpreter to run. There are embedded scripting languages that you cannot run this way.

@SeeminglyScience

That can already be done with here-strings right? Can you elaborate a little bit on what benefits this provides over existing methods?

People can use here-strings today or they can escape all the arguments passed to native commands if they can figure out how to get it right. The intent here is to make it simpler for new users for cut-and-paste (Stackoverflow) scenarios.

Also why stop at python and bash? Why not C#, CIL, C++?

The scenarios being supported here is a block of text passed to a native command. Any language can be supported if it supports that calling convention. There is no proposal to create a temp source file and have it be compiled.

I'm not against this because I think I'm going to be forced to use it. I don't like it because it's going to be difficult to maintain from a language perspective, a nightmare for editors, and ultimately encourage scripts that are unreadable unless you know several languages.

There is no expectation that you get mixed language syntax coloring. Any nested script from another language will just be monotoned.

Really though this is a pretty different concept from the original topic of this thread with much wider implications. I'd recommend creating a new issue for it.

The proposed implementation is different from the original issue, but the problem is the same which is the cut-and-paste and "just work" scenario.

People can use here-strings today or they can escape all the arguments passed to native commands if they can figure out how to get it right. The intent here is to make it simpler for new users for cut-and-paste (Stackoverflow) scenarios.

Right but how is it actually easier? I get that it's subjective but these don't seem that different to me:

~~~powershell
$a = ```bash
find . -iname *$pdf -print0 | xargs -0 ls -ltr

~~~

vs

```powershell
$a = bash -c @'
   find . -iname *$pdf -print0 | xargs -0 ls -ltr
'@

They seem almost the same with the exception that the latter is significantly more clear in what is going to happen.

The scenarios being supported here is a block of text passed to a native command. Any language can be supported if it supports that calling convention. There is no proposal to create a temp source file and have it be compiled.

Well C# wouldn't need a temp source file. Maybe not CIL either (no idea about C++). Either way though a code fence strictly being syntactic sugar for calling an executable (e.g. bash/python/cmd) is confusing to me. Code fences imply language support imo.

There is no expectation that you get mixed language syntax coloring. Any nested script from another language will just be monotoned.

It'll still be a nightmare for tmLanguage based parsers. They can barely handle syntax they expect atm. More than that though, if that's the case I don't get why it's different from a here-string.

The proposed implementation is different from the original issue, but the problem is the same which is the cut-and-paste and "just work" scenario.

Fair but we're already 145 comments into discussing the previously proposed solutions. Especially since this thread already came to a conclusion, you'll likely see a lot more involvement in a new, dedicated thread.

Well C# wouldn't need a temp source file. Maybe not CIL either (no idea about C++). Either way though a code fence strictly being syntactic sugar for calling an executable (e.g. bash/python/cmd) is confusing to me. Code fences imply language support imo.

I just want to echo @SeeminglyScience's comment on this. We can run C# script without involving compilation with the library Microsoft.CodeAnalysis.CSharp.Scripting. That's what dotnet-interactive uses in the C# Jupyter kernel. So it might not be unreasonable for users to ask for C# scripting or even F# scripting support with such a syntax.

You can say that the code fencing syntax in PowerShell is for calling a native command, which theoretically allows this:

    ```c:\mypath\MyArbitraryExe
        args to my arbitrary executable
    ```

but then it's different from users' established understanding of it from markdown, which, as @SeeminglyScience called out, implies language support. Using a similar syntax with different semantics is confusing in my opinion.

which theoretically allows this

You put the script inside, not the arguments, and the script is equivalent to the input stream. Also, if this is meant to provide SO support, full paths should not be allowed.

You put the script inside, not the arguments, and the script is equivalent to the input stream

The script is the argument for bash/cmd/python.

Also, if this is meant to provide SO support, full paths should not be allowed.

Then switch it for this example:

~powershell
$a = ipconfig /all
~

Or add the arbitrary exe to path first.

You put the script inside, not the arguments, and the script is equivalent to the input stream

The script is the argument for bash/cmd/python.

None of them allows to pass a script as a positional argument. CMD does not even allow a batch file.

Also, if this is meant to provide SO support, full paths should not be allowed.

Then switch it for this example:

$a = ```ipconfig
/all

That would not work.

Seems like we'd be making an awful lot of special cases on very arbitrary basis for this markdown idea to work.

Also, the main reason this is even supported in Markdown is for syntax highlighting. That's going to be a constant request if we add something like this into the language.

This is not an effective solution.

Was discussing with @jpsnover about this and he proposed the shebang #! as the sigil

just to add my 2 cents, i think it's a mistake to restart this discussion after a solution had been announced. and personally i dislike both the shebang and markdown syntax (and both are breaking changes).

I don't mind restarting discussions if someone has come up with something demonstrably better. Indeed better to do that now than after the feature has shipped. I just don't think #! or the markdown approach is a better solution to supplying an "un-PowerShell-adulterated" command line string to a native exe. Maybe I just need to try it to get used to it but my initial reaction is ... um, eew.

_If we (and PowerShell) know that we run a native executable why do we need "call native operator" at all?_

If PowerShell parse an input string and get CommandAst then PowerShell does a discover of the command.
If the command is a native command PowerShell converts the ast to NativeCommandProcessor.
We can implement new (experimental NativeCommandProcessor) for the case which will re-parse the Ast extent (that is the input string) so that to get the native executable name/path and rest of the string as argument or arguments by OS rules.

If an user want copy-paste from shell he should type "cmd " or "bash " - that will works without editing.
If an user want copy-paste an native command line will works too like g++ timer.cpp @conanbuildinfo.args -o timer --std=c++11.

If an user want copy-paste from shell

We have Get-/Set-Clipboard for that.

for the case which will re-parse the Ast extent (that is the input string) so that to get the native executable name/path and rest of the string as argument or arguments by OS rules.

My thinking here is that doing this you either:

• Inefficiently parse the command AST, only to reparse the extent later as a string and throw the original AST away, OR
• You take the AST and try to reform it back into a string (the way the current NativeCommandProcessor does), but in doing so things will be lost

So instead we need to preserve the string from the start. Now, I think arguably PowerShell already has three kinds of string token, but the main request in this discussion seems to be not wanting to escape things. Any string needs to be able to escape its terminator, but the solution discussed there seems to be using newline as the terminator and not being able to escape newlines (there's always semicolons).

I'm not totally wedded to a newline-only terminator, but it does seem to be the only solution to not wanting to escape anything. Basically that would make a "native call operator" tokenise similarly to how a line comment is today — from the operator to the end of the line is the string passed to the "native shell".

So:

--% ps -o pid,args -C bash | awk '/running_script/ { print $1 }'

will tokenise as:

--% ps -o pid,args -C bash | awk '/running_script/ { print $1 }'
|-||-----------------------------------------------------------|
 |                                  \
Native call operator        Invocation to be passed to native shell

(Note that we send even the space directly after --% to the native shell, since the string begins immediately after --%)

Then this token is wrapped in a very simple AST, like:

// We need to carefully work out what AST this inherits from
// This syntax has almost no interoperability with PowerShell by design,
// so can't implement fields like redirections or backgrounding faithfully.
// But in order to participate in pipelines and similar, would need to extend a more concrete class
public class NativeCallInvocationAst : StatementAst
{
    public string NativeInvocation { get; }
}

Then, after this is compiled and sent to the pipe, this uses a new native command processor to send the string directly as an argument to the native shell.

While that all seems implementable, some questions in my mind are:

• Is the native shell configurable? If not, how do we justify that? If so, should it be explicit as part of the syntax (complicating the syntax), or implemented as a preference variable (making it harder to discover and manage)?
• Is such an implementation going to be discoverable and not confuse people? Will users understand how little PowerShell is doing here, or what's happening when they use | or & or even strings?
• How many scenarios are going to be serviced by a single-line verbatim string here? Are we going to implement this just to find out that many people want multi-line invocations? Is the need to avoid doing something like escaping a single quote great enough to add these constraints?
• This seems like a very specific scenario for a whole new language syntax. If we're creating a new verbatim string syntax, why are we coupling it directly to the concept of native invocation? If we're creating a new invocation mechanism, why are we coupling it directly to the concept of other shells? Good syntax and good language constructs compose, both syntactically and functionally, and try not to bake too much into the language itself (instead providing a platform of elements for the user to assemble as their program) — does our proposed native call syntax meet the bar for a new elementary syntax to be composed into programs? Is there a better alternative approach we could take that solves this problem in a simple way, but still allows us to keep the building blocks uncoupled so they can be composed to solve other problems?
• Would other shells or other languages implement this or have they already? At a syntax level, or at any other level? If so, how?

My thinking here is that doing this you either

@rjmholt CommandAst has Extent property with input string. Re-parsing is as simple as split by first space on Windows or split by spaces on Unix. So I expect we lost nothing in the string and get good performance.

However this gets implemented, I think we need a set of use cases to test against. Let me "propose" the following:

• Use case 1: I want a single exe literal (no interpolation) invocation to work without knowing about quoting/escaping rules e.g.: g++ timer.cpp @conanbuildinfo.args -o timer --std=c++11. The current work-around for this is to figure out what to quote e..g: g++ timer.cpp '@conanbuildinfo.args' -o timer --std=c++11 A more complicated example might include command lines spanning multiple lines e.g.:

tar -cvpzf /share/Recovery/Snapshots/$(hostname)_$(date +%Y%m%d).tar.gz \
    --exclude=/proc \
    --exclude=/lost+found 

Current workaround (swap backtick for \) e.g.:

tar -cvpzf /share/Recovery/Snapshots/$(hostname)_$(date +%Y%m%d).tar.gz `
    --exclude=/proc `
    --exclude=/lost+found

• Use case 1a: I want a single exe invocation with PS interpolation to work e.g.: g++ $CppFile @conanbuildinfo.args -o timer --std=c++11. The current work-around for this is to figure out what to quote (single or double) and what to escape e..g: g++ $CppFile '@conanbuildinfo.args' -o timer --std=c++11 A more complicated example might include command lines spanning multiple lines e.g.:

tar -cvpzf "/share/Recovery/Snapshots/${ComputerName}_`$(date +%Y%m%d).tar.gz" \
    --exclude=/proc \
    --exclude=/lost+found 

Current workaround, double-quote first arg, swap backtick for \ and escape the $ at the start of `$(date + ...).

• Use case 2: I want to execute a literal pipelined command in another shell e.g.: ps -o pid,args -C bash | awk '/running_script/ { print $1 }' and wsl ls $foo && echo $PWD. _I'm not sure if this needs to be a separate use case from the previous one._ The current work-around is to quote the bash command e.g.: bash -c 'ps -o pid,args -C bash | awk ''/-bash/ { print $1 }'''. This use case might include multiline strings as well e.g.:

ps -o pid,args -C bash | \
awk '/running_script/ { print $1 }'

• Use case 2a: I want to execute a pipelined command in another shell with some PS interpolation - is this a use case?

• Use case 3: I need npm run $scriptName to continue to work i.e. interpolate variables. (this may be implicit and not need to be stated).

There's a question of whether there should be use cases for interpolating strings i.e. g++ $CppFile @conanbuildinfo.args -o timer --std=c++11 or do we just punt on that and say that the current escaping rules cover this?

These are likely not the right (or all) use cases but I think it would be helpful to have documented use cases and be able to refer to them when discussing this feature otherwise it's easy to get lost in how the feature actually addresses (or not) customer use cases.

Also, the second use case seems to me to require "shell invocation". I'm not sure the first one does though. Seems like the exe could be invoked directly.

@rkeithhill Could you add existing workarounds for completeness in your previous post?

@oising, I too appreciate your thoughtful response.

What does ins ... give that & cmd --% ... cannot?

--% in its current, _parameter_ (symbol) form - whose behavior I presume we want to retain - cannot be used for cmd /c --%, because it doesn't support _multi-command_ cmd command lines (nor ^ as the escape char.), given that the first unquoted | will be interpreted as _PowerShell's_ pipe operator and that & is passed verbatim.
The same would apply on Unix, where --% is generally near-useless, and with sh -c fails altogether, because sh expects the command line as a _single_ argument (e.g., sh -c --% echo hi yields an empty line, because echo alone is executed as the command).

ins, with its requirement to delimit the pass-through command line by passing it _as a single string_ solves these problems, and, as stated, also allows the use of string interpolation to incorporate _PowerShell_ variable and expression values.

And the latter brings us to why the proposed --% _statement_ is a _dead-end street_:

• There is _no transition path_ from executing a preexisting command line _exactly as is_ to incorporating _PowerShell_ variable and expression values into it.

  • The only - obscure and cumbersome - way to achieve this is to define _auxiliary environment variables_ that store the PowerShell values of interest and that you can then reference (shell-appropriately with %...% or $...) in the native command line.

• You have the awkwardness of not being able to incorporate a --% statement into a larger PowerShell pipeline.

  • Note that there are legitimate _non_-cut-and-paste reasons to call the native shell, where you may expect this integration; specifically, passing _raw byte data_ through (an intermediate) pipeline and sending _strings without a trailing newline_ require use of the native pipe - see this SO answer for a real-life example.

• There is also the awkwardness of the --% _parameter_ being virtually useless on Unix, which introduces an odd asymmetry: if the --% _statement_ works on Unix too, why not the --% _parameter_? (e.g.,
  /bin/echo --% 'hi "noon"' yields verbatim 'hi noon' (rather than the expected hi "noon"), due to echo receiving _two_ arguments, verbatim 'hi and noon', with single quotes retained as data and double quotes stripped).

ins, as the suggested alternative to the --% statement, would ideally just be a _convenience wrapper_ around cmd /c and sh -c, but it is actually _required_:

• At least _temporarily_ on Unix, as long as #1995 isn't fixed, because calling sh -c '...' directly currently doesn't pass arguments with embedded " through properly.

• _Invariably_ on Windows, because you need to pass the command line _as-is_ to cmd via the lpCommandLine parameter of CreateProcess (which the --% parameter cannot do due to being limited to _one_ command); alternatively trying to pass the command line as a _single string_ enclosed in "..." overall again doesn't work robustly due to #1995.

The troubles with _argument_ passing could be avoided by providing the command line to execute _via stdin_ (which ins could and should also support), i.e. to _pipe_ the command line / script text to the shell executable:

• On Unix, this works just fine, because passing code via _stdin_ to sh is essentially the same as passing a _script file_ for execution (just as sh -c '<code>' is):

PSonUnix> @'
echo '{ "foo": "bar" }' | cat -n
'@ | sh

     1  { "foo": "bar" }

• Alas, on Windows cmd doesn't handle such input nicely: it prints its logo and echoes each command before execution, along with the prompt string:

PSonWin> 'date /t & ver' | cmd

Microsoft Windows [Version 10.0.18363.836]
(c) 2019 Microsoft Corporation. All rights reserved.

C:\Users\jdoe>date /t & ver
Sun 07/26/2020

Microsoft Windows [Version 10.0.18363.836]

C:\Users\jdoe>

Sadly, PowerShell itself exhibits similarly unhelpful behavior: see #3223

Just like sh and all major POSIX-like shells (dash, bash, ksh, zsh), most scripting languages _do_ properly support stdin-as-script input, such as python, node, ruby, and perl.

PS> 'print("hi")' | python

hi

For incorporating _PowerShell_ variable and expression values, you again have the option of using string interpolation / concatenation:

PS> $foo = 'bar'; "print(`"$foo`")" | python

bar

or using the interpreter's argument-passing feature:

PS> @'
import sys
print(sys.argv[1])
'@ | python - bar

bar

I hope the above makes it clear that there is _no_ need for special syntax at all - neither a --% statement, nor a #! statement, nor Markdown code blocks - all of which again would lack the ability to incorporate values from the calling PowerShell script.

As for the configurability of the native shell to use: For predictability, I don't think we should offer one; users who want to target a different shell can just pipe the command line to that shell's specific executable or, once #1995 is fixed, pass it as an argument.

• A debatable variation is to target /bin/bash (with an (unlikely) fallback to /bin/sh) - while one would hope that published command lines rely on POSIX-mandated features only, command lines with Bash-isms (non-POSIX features that not all /bin/sh implementations can be expected to support) are likely out there, given the prevalence of bash.

@joeyaiello

It's my view that the benefits of this particular feature are much more significant than the impact #1995.

I find it mind-boggling that we spend as much effort as we have on making this cut-and-paste feature - primarily for _interactive_ convenience - an (awkward, feature-limited) first-class citizen, whereas PowerShell's fundamental inability to pass empty arguments and arguments with embedded " chars. to external programs has been left unaddressed for years.

but I'm not seeing people hit #1995 in the wild en masse

I do, and for the reasons I've previously discussed _you'll see it more and more_.

Again, I would argue a shell that does the following, as PowerShell currently does, has a serious problem:

# On Unix
PS> /bin/echo '{ "foo": "bar" }'

{ foo: bar }

@rjmholt CommandAst has Extent property with input string. Re-parsing is as simple as split by first space on Windows or split by spaces on Unix. So I expect we lost nothing in the string and get good performance.

Yeah, I understand the draw there, and perhaps the allocations of the AST in that scenario aren't awful (although I'd argue that the language ought not to generate needless allocations for simple syntax). But all you have to do is imagine an input that bash/cmd sees as a string and PowerShell doesn't to start hitting issues:

Does not parse with current PowerShell:

--% my_command "\" "

Parses, but splitting by spaces gives the wrong result:

--% my_command "\"    "\"

(We must send 4 spaces in the string, but PowerShell sees two strings separated in an array)

@rjmholt, all great questions at the bottom of your comment above, which to me again suggest that the right answer is: no need for special syntax.

I've addressed the question about what native shell executable to use in my previous comment, but as for:

Would other shells or other languages implement this or have they already? At a syntax level, or at any other level? If so, how?

I am not aware of any other language having done this at the syntax level without employing _explicit delimiters_ and allowing _interpolation_.
E.g., perl has `...` for running shell commands, so you can run the following command from a script on Unix, for instance:
my $foo='/'; print `ls $foo | cat -n`

The key aspects are the use of explicit delimiters and the ability to incorporate values from the caller - both of which are missing from the current proposal for the --% statement.

As stated before, we _could_ use this approach for a new _operator_ (or some sort of delimiter-based, optionally interpolating syntax), but I don't think that's worth it, given that
ins "ls $foo | cat -n" (or with a here-string variant) will do, without burdening the language with additional complexity.

perl offers a (proper) syntax-level solution _because it isn't itself a shell_ but wants to make shell calls as convenient as possible.

By contrast, we _are_ a shell (too), and offer shell-style invocations _directly_, without any additional syntax (albeit presently flawed - see #1995).
Providing special syntax to make _a different shell's_ invocations as easy as possible seems unnecessary.

@iSazonov

_If we (and PowerShell) know that we run a native executable why do we need "call native operator" at all?_

If PowerShell parse an input string and get CommandAst then PowerShell does a discover of the command.
If the command is a native command PowerShell converts the ast to NativeCommandProcessor.
We can implement new (experimental NativeCommandProcessor) for the case which will re-parse the Ast extent (that is the input string) so that to get the native executable name/path and rest of the string as argument or arguments by OS rules.

If an user want copy-paste from shell he should type "cmd " or "bash " - that will works without editing.
If an user want copy-paste an native command line will works too like g++ timer.cpp @conanbuildinfo.args -o timer --std=c++11.

But wait wouldn't you never be able to use any PowerShell syntax with native commands if we did all that? So like sc.exe query $serviceName would pass $serviceName as a literal string?

But wait wouldn't you never be able to use any PowerShell syntax with native commands if we did all that? So like sc.exe query $serviceName would pass $serviceName as a literal string?

The --% option is not intended to address that need.

sc.exe query $serviceName is handled by &"<cmd>"

The --% option is not intended to address that need.

sc.exe query $serviceName is handled by &"<cmd>"

Yeah for sure, I don't think that's what @iSazonov is referring to though. I'm reading that as though they're suggesting a general change to how parameter binding works for native commands.

Yeah for sure, I don't think that's what @iSazonov is referring to though. I'm reading that as though they're suggesting a general change to how parameter binding works for native commands.

Lord, I hope not.

Lord, I hope not.

Eh sometimes when you're brainstorming the abstract it's easy to forget about the obvious. I mean to be fair we all read it and immediately started thinking implementation without realizing how much it would break.

Or I'm missing something obvious and that's not at all what's being proposed 😁

@essentialexch, to be clear, re:

sc.exe query $serviceName is handled by &"<cmd>"

& "<cmd>" only works if <cmd> evaluates to a mere command _name or path_, _not including arguments_ - arguments must be passed separately, individually - and they don't per se need "...." quoting for variable references to be expanded (e.g.,
$exe = 'findstr'; & "where.exe $exe" fails by design; must be & "where.exe" $exe (double quotes optional here; if omitted, the & is then optional too))

Yeah for sure, I don't think that's what @iSazonov is referring to though. I'm reading that as though they're suggesting a general change to how parameter binding works for native commands.

My proposal is to fix #1995 (with breaking change) as simple as possible. Of cause interpolation does not work. Of cause it is not final proposal - it demo how we could implement a fix in minutes. And it mainly demos that we do not need a call native operator at all as @mklement0 pointed too.

Parses, but splitting by spaces gives the wrong result:

@rjmholt Yes, but I think you catch the idea.
First child of the CommandAst is StringConstantExpressionAst with the native command as BareWord. We can cut the bare word from CommandAst Extent and get parameter list without ast-s change. But we could add new ast to keep the parameter list too.
I spend a time for the explanation because below I want to summarize the discussion and I feel we will need to implement some enhancement which looks complex but I feel they could be implemented simple and __with easy opt-out for backward compatibility__.

I believe the @SteveL-MSFT's intention of the initial proposal was to avoid a breaking change.
I agree with @TSlivede and @mklement0 that this discussion showed that the new syntax is not needed since it does not solve all _fundamental_ problems like #1995 and #12975. On the other hand, it adds complexity to the language, while we want, on the contrary, to _simplify_ work with native applications.

__The only way to move forward is to make a breaking change or more.__ (After that, a new operator or cmdlet might be useful in some scenarios too.)

We should start with summarization of use cases and user expectations.
See @rkeithhill's great post above

1. Users want enable/disable interpolation.
   PowerShell already has strings/here-strings with single and double quotes.
   Our goal is to understand how to apply/refine/enhance its.

2. Users want run any executable or a shell

   • any executable - command starts with the app name and follows arguments
     
     
     
     • with/without interpolation
     
     
     • on single line/on multi line
     
     
   • shell is probably a special case because argument(s) is a _script_
     
     
     
     • in copy-paste scenario users don't want interpolation and want single and multi line
     
     
     • in script scenario users want to enable/disable interpolation and want single and multi line
     
     

   Thoughts:
   This makes us think that the copy-paste scenario behavior should be the default.
   Line terminators depend on executable ( in bash, ` in PowerShell). This makes us think that:
   (1) a shell should be explicitly stated as any executable,
   (2) perhaps any executable with multi line should be called by means of a shell or we have to do a special parsing (to remove terminator and build argument list falling back to the issue with escaping)
   (3) we could address copy-paste scenario in PSReadline. It could convert a buffer to right PowerShell command.

3. Users want single line and multiline.

   • perhaps we need to enhance here-strings for single line to address some scenarios as discussed previously.
   • see 2 - (2) - always use a shell for multiline or implement a special parsing

I don't want to write more because @TSlivede and @mklement0 could update their thoughts and conclusions from earlier but which can now be based on acceptance of breaking change(s). I'd ask to open new issue (and close all olds), enumerate all use cases (starting with @@rkeithhill's and add more), and maybe create Pester tests - it would be great step to fix the Issue.

I want only add that we could consider new cmdlets to simplify user adoptions like Invoke-Shell (Invoke-NativeShell), Test-Arguments (like echoargs.exe to show that native app gets and show a help for users), Convert-Arguments (to convert user input to right escaped arguments).

PS: As I showed above single line behavior could be easily opt-out for backward compatibility at run time.

Line terminators depend on executable ( in bash, ` in PowerShell).

\ is not a line terminator in bash.

Of cause interpolation does not work. Of cause it is not final proposal - it demo how we could implement a fix in minutes. And it mainly demos that we do not need a call native operator at all as @mklement0 pointed too.

In the same way that we could fix all bugs in a few minutes if we just deleted the repo. That's a pretty enormous break you're proposing. Even if just a hypothetical I don't see how it's helpful.

Even if just a hypothetical I don't see how it's helpful.

@SeeminglyScience It seems we are in different contexts. I say that we can implement a fix for single line native call as a breaking change but without breaking Parser. In other words, we can even switch the behavior on the fly, no matter what new behavior we would implement.

In other words, we can even switch the behavior on the fly, no matter what new behavior we would implement.

I guess the thing I'm missing is how do you switch that behavior on the fly without explicitly asking for it (like with a call native operator).

@iSazonov

_If we (and PowerShell) know that we run a native executable why do we need "call native operator" at all?_

If PowerShell parse an input string and get CommandAst then PowerShell does a discover of the command.
If the command is a native command PowerShell converts the ast to NativeCommandProcessor.
We can implement new (experimental NativeCommandProcessor) for the case which will re-parse the Ast extent (that is the input string) so that to get the native executable name/path and rest of the string as argument or arguments by OS rules.

If an user want copy-paste from shell he should type "cmd " or "bash " - that will works without editing.
If an user want copy-paste an native command line will works too like g++ timer.cpp @conanbuildinfo.args -o timer --std=c++11.

I want to confirm what you mean by that: Are you suggesting to essentially ignore all powershell syntax, when calling external executables?

Specifically: Are you suggesting, that running for example

/bin/echo (1..5)

in powershell will no longer out put 1 2 3 4 5 but should instead output literal (1..5)?

If that is actually what you are suggesting, than I have to say: I think that is a terrible idea.

That doesn't simplify things (external executables behave now syntactically different than internal cmdlets) and it also does't have to do anything with #1995 IMHO...

@rjmholt
From above:

• Would other shells or other languages implement this or have they already? At a syntax level, or at any other level? If so, how?

The only thing, that comes to my mind would be ipythons ! operator:

Running !ps -'fax' || true in ipython outputs:

  261 pts/0    Sl     0:00          \_ /usr/bin/python /usr/bin/ipython
  311 pts/0    S      0:00              \_ /bin/bash -c ps -'fax' || true
  312 pts/0    R      0:00                  \_ ps -fax

(ps doesn't show quotes around arguments, but ps -'fax' || true is really given as one single argument to bash.)

However, this feature still allows interpolation of python variables into that command ({pyvar}).

And similar to @mklement0's example, ipython only implements this

because it isn't itself a shell

If ipython would only allow python syntax it wouldn't be very useful as a shell, so they allow this syntax to forward commands to bash to be usable as a shell. Powershell on the other hand claims to be a shell (until #1995 is solved, I won't say that it is a shell), so it would be quite strange to add a special syntax for calling other shells.

If something like this gets for whatever reason actually implemented (hopefully not), I'd suggest to use ! instead of --% as the "call shell operator".

! is something people might guess (because they know ! for that purpose from ipython (or from the vim command mode or from less or from ftp))

--% on the other hand is harder to type, unlikely to be guessed and most importantly: The current usage of --% has in my opinion very little in common with the "pass stuff to another shell" behavior. The current --% emulates exactly one element of cmd syntax: variable substitution (and even that not completely). It doesn't support the escape char ^, doesn't allow redirection to files, and so on. The only thing somewhat useful about the current --% is the ability to pass verbatim content into the lpCommandLine - but that is something that the proposed "call shell operator" isn't good at.

One more difference (which was already mentioned) between the proposed "call shell operator" and the current --%: One ends at pipes, the other doesn't - very confusing for new users. ! on the other hand forwards | to the shell in all applications I tested so far (ipython, vim, less, ed, etc.).

HELP about_Logical_Operators -S

Yeah, ! by itself is problematic because it's a logical negation operator in the language.

BTW one of the alternate proposals was &! - sort of a "call native/shell" operator.

I wish @TSlivede had stressed the "hopefully not" part more when he said "If something like this gets for whatever reason actually implemented (hopefully not)"

• No operator/statement _with individual arguments_ can cleanly solve the I-want-to-use-PowerShell-values-in-the-native-command-line problem, given that $ is used as the variable sigil in _both_ PowerShell and POSIX-like shells.

• No operator/statement _without explicit delimiters around the entire native command line_ can solve the where-does-the-native-command-line-end problem (which you may or may not care about).

@PowerShell/powershell-committee discussed this today. At this point, there doesn't seem to be agreement on the initial problem statement nor a design on how to resolve so we don't believe we can get this into 7.1 w/ a stable design.

@PowerShell/powershell-committee discussed this today. At this point, there doesn't seem to be agreement on the initial problem statement nor a design on how to resolve so we don't believe we can get this into 7.1 w/ a stable design.

182 comments!

There's been some great discussion on this! I'll continue to monitor any new comments on this issue. We still encourage the community to implement Invoke-NativeCommand type cmdlet published to PSGallery independent of this issue.

I think it unfortunate, when you had a compromise but very useful solution at https://github.com/PowerShell/PowerShell/issues/13068#issuecomment-662732627, to drop this. I know you want consensus, but sometimes a compromise is the best that can happen. No, it isn't perfect, but a one-liner SO solution would be extremely useful. At least in this one person's opinion.

@essentialexch to be clear, we don't even have consensus within the PowerShell team

Since this is postponed what about implementation of helpers

we could consider new cmdlets to simplify user adoptions like Invoke-Shell (Invoke-NativeShell), Test-Arguments (like echoargs.exe to show that native app gets and show a help for users), Convert-Arguments (to convert user input to right escaped arguments).

I think Test-Arguments could be very useful.

I've just published a module, Native, (Install-Module Native -Scope CurrentUser) which I invite you all to try and whose commands I'll be using below; the numeric use cases referenced are from @rkeithhill's comment above.

If we ever want the conceptual fog to clear, I think we need to frame the uses cases differently.
_None of them require any changes in parsing_.

Use case [native-shell call]:

You want to execute an _individual command_ (use case 1) or an entire _multi-command command line_ (use case 2) _written for the platform-native shell_ (this issue):

• Since you're using a different shell's syntax, you're passing the command (line) _as a single string_ to the target shell, for _it_ to do the parsing; PowerShell's string interpolation offers the ability to embed PowerShell values into the string passed (use case 2a).
• ins (Invoke-NativeShell) covers these uses cases.

# Use case 1: Single executable call with line continuation, on Unix.
@'
tar -cvpzf /share/Recovery/Snapshots/$(hostname)_$(date +%Y%m%d).tar.gz \
    --exclude=/proc \
    --exclude=/lost+found 
'@ | ins

# Use case 2: Entire Bash command line (also with line continuation)
@'
ps -o pid,args | awk \
  '/pwsh/ { print $1 }'
'@ | ins

# Use case 2a: Entire Bash command line (also with line continuation) with string interpolation.
# Note the double-quoted here-string and the need to escape the $ that is for Bash as `$
$fields = 'pid,args'
@"
ps -o $fields | awk \
  '/pwsh/ { print `$1 }'
"@ | ins

# Alternative to use case 2a: pass the PowerShell value *as a pass-through argument*,
# which allows passing the script verbatim.
# Bash sees the pass-through arguments as $1, ... (note that the `awk` $1 is unrelated).
@'
ps -o $1 | awk \
  '/pwsh/ { print $1 }'
'@ | ins - 'pid,args'

The here-string syntax isn't the most convenient (hence the suggestion to implement an in-line variant - see #13204), but you don't _have_ to use it; for verbatim commands, you can use '...', which only requires _doubling_ embedded ', if present.

Also, here's a reasonable substitute for the "StackOverflow operator":

If you place the following PSReadLine key handler in your $PROFILE file, you'll be able to use Alt-v to scaffold a call to ins with a verbatim here-string into which the current clipboard text is pasted.
Enter submits the call.

# Scaffolds an ins (Invoke-NativeShell) call with a verbatim here-string
# and pastes the text on the clipboard into the here-string.
Set-PSReadLineKeyHandler 'alt+v' -ScriptBlock {
  [Microsoft.PowerShell.PSConsoleReadLine]::Insert("@'`n`n'@ | ins ")
  foreach ($i in 1..10) { [Microsoft.PowerShell.PSConsoleReadLine]::BackwardChar() }
  # Comment the following statement out if you don't want to paste from the clipboard.
  [Microsoft.PowerShell.PSConsoleReadLine]::Insert((Get-Clipboard))
}

Use case [direct executable call]:

You want to call a _single external executable_ using _PowerShell_'s syntax, with _individual arguments_ (use cases 1a and 3).

• This is a core mandate of any shell, and it is of vital importance that it work robustly.

  • You need to be able to rely on PowerShell being able to pass through any arguments that result from _its_ parsing _as-is_ to the target executable. This is currently not the case - see #1995.

• It requires you to understand _PowerShell's_ syntax and _its_ argument-mode metacharacters. Of necessity, these differ from the native shells', but it is the price to pay for PowerShell's superior command-line capabilities - it's a price we want to encourage users to pay.

  • You need to be aware that ` is used as the escape character and for line continuation.

  • You need to be aware that PowerShell has additional metacharacters that require quoting/escaping for verbatim use; these are (note that @ is only problematic as an argument's first char.):

    • for POSIX-like shells (e.g., Bash): @ { } ` (and $, if you want to prevent up-front expansion by PowerShell)
    • for cmd.exe: ( ) @ { } # `
    • Individually `-escaping such chars. is sufficient (e.g., printf %s `@list.txt).

While we're waiting for #1995 to be fixed, function ie (for invoke (external) executable) fills the gap, simply by prepending to a direct call; e.g., instead of the following call:

# This command is currently broken, because the '{ "name": "foo" }' argument isn't properly passed.
curl.exe -u jdoe  'https://api.github.com/user/repos' -d '{ "name": "foo" }'

you'd use the following:

# OK, thanks to `ie`
ie curl.exe -u jdoe  'https://api.github.com/user/repos' -d '{ "name": "foo" }'

The Native module comes with something akin to echoArgs.exe, the dbea (Debug-ExecutableArguments) command; sample output on Windows:

# Note the missing first argument, the missing " chars., and the erroneous argument boundaries.
PS> dbea '' 'a&b' '3" of snow' 'Nat "King" Cole' 'c:\temp 1\' 'a \" b'
7 argument(s) received (enclosed in <...> for delineation):

  <a&b>
  <3 of snow Nat>
  <King>
  <Cole c:\temp>
  <1\ a>
  <">
  <b>

Command line (helper executable omitted):

  a&b 3" of snow "Nat "King" Cole" "c:\temp 1\\" "a \" b"

By using the -UseIe (-ie) switch, you can tell dbea to pass the arguments via ie, which demonstrates that it fixes the problems:

# OK, thanks to -UseIe
PS> dbea -UseIe '' 'a&b' '3" of snow' 'Nat "King" Cole' 'c:\temp 1\' 'a \" b'
6 argument(s) received (enclosed in <...> for delineation):

  <>
  <a&b>
  <3" of snow>
  <Nat "King" Cole>
  <c:\temp 1\>
  <a \" b>

Command line (helper executable omitted):

  "" a&b "3\" of snow" "Nat \"King\" Cole" "c:\temp 1\\" "a \\\" b"

Note: Once #1995 is fixed, ie is no longer required; in the interest of forward compatibility, ie is designed to detect and automatically defer to a fix; that is, once the fix is in place, ie will stop applying its workarounds and will effectively act like a direct / & call.

Use case [direct rogue Windows executable call]:

There are two main problems, which arise on _Windows only_:

• You're invoking a "rogue" executable that has quoting requirements that differ from the most widely used convention; e.g, msiexec.exe and msdeploy.exe require prop="<value with spaces>" to be quoted precisely that way - the quoting just around the _value_ part - even though "prop=<value with spaces>" - quoting of the entire argument - _should_ be equivalent (the latter is what PowerShell - justifiably - does behind the scenes).

• You're invoking _a batch file_ with _an argument that doesn't contain spaces, but contains cmd.exe metacharacters_ such as &, ^, or |; e.g.:

  • .\someBatchFile.cmd 'http://example.org/a&b'

  • PowerShell - justifiably - passes http://example.org/a&b _unquoted_ (since there is no embedded whitespace), but this breaks the batch-file invocation, because cmd.exe - unreasonably - subjects the arguments passed to a batch file to the same parsing ruling that would apply inside of cmd.exe rather than accepting them as literals.

  • Note: While using batch files to _directly_ implement functionality is probably waning, using them as _CLI entry points_ is still very common, such as Azure's az CLI, which is implemented as batch file az.cmd.

Note: ie automatically handles these scenarios for batch files and for msiexec.exe and msdeploy.exe, so for _most_ calls no extra effort should be needed; however, it is impossible to anticipate _all_ "rogue" executables.

There are two ways to resolve this:

• Given that cmd.exe, after applying its own interpretation to arguments on a command line, _does_ preserve the quoting as specified, you can delegate to an ins call on Windows:

  • ins '.\someBatchFile.cmd "http://example.org/a&b"'
  • If you use an _expandable_ string, this again enables you to embed PowerShell values into the command string.
    
    
    
    • $url = 'http://example.org/a&b'; ins ".\someBatchFile.cmd `"$url`""
    
    

• Alternatively, use the current --% implementation, but beware its limitations:

  • .\someBatchFile.cmd --% "http://example.org/a&b"'
  • Given how --% is implemented, the only way to embed PowerShell values is to - awkwardly - define an _aux. environment variable_ and reference it with %...% syntax:
    
    
    
    • $env:url = 'http://example.org/a&b'; .\someBatchFile.cmd --% "%url%"
    
    

Error handling

The pending https://github.com/PowerShell/PowerShell-RFC/pull/88 will bring better error-handling integration with external (native) executables.

In the meantime, the Native module ships with two features for ad-hoc opt-in to treating a nonzero exit code as a script-terminating error:

• ins supports the -e / -ErrorOnFailure switch, which throws an error if $LASTEXITCODE is nonzero after the call to the native shell.

• iee is a wrapper function for ie that analogously throws an error if $LASTEXITCODE is nonzero after the call to the external executable.

There are many subtleties to the commands that ship with the module.
The fairly extensive comment-based help hopefully covers them sufficiently.

If we ever want the conceptual fog to clear, I think we need to frame the uses cases differently.

Great! What I posted earlier was just to get folks thinking about creating more and better use cases.

I should clarify a few important aspects of ins (Invoke-NativeShell) that differ from what I've previously proposed; the aim was to account for the ubiquity of bash and to provide a consistent CLI across platforms.

• On Unix, I've decided to call /bin/bash rather than /bin/sh, given Bash's ubiquity, but you can opt into using /bin/sh with -UseSh (-sh) (/bin/sh also serves as the fallback in the unlikely event that /bin/bash isn't present).

  • In the interest of a consistent experience across invocation forms and platforms, I've hidden the ability to set $0, the invocation name; that is, any pass-through arguments start with $1, which is consistent with how arguments are passed when you pipe the script from the pipeline, and likely what users expect:

# Script as argument
PSonUnix> ins 'echo $# arguments; echo "[$1] [$2]"' one two
2 arguments
[one] [two]

# Script via pipeline; note the "-" to signal that the script is piped.
PSonUnix> 'echo $# arguments; echo "[$1] [$2]"' | ins - one two
2 arguments
[one] [two]

# As an aside: script as argument, with pipeline input *as data*:
PSonUnix> 'one', 'two' | ins 'cat -n'
     1  one
     2  two

• On Windows I had to use a temporary _batch file_ behind the scenes for technical reasons, but I think that use of batch-file syntax is ultimately the better choice anyway (%%i rather than %i in for loop variables, ability to escape % as %%).

  • Use of a batch file also enables support for pass-through arguments, as on Unix:

# Script as argument
PSonWin> ins 'echo [%1] [%2]' one two
[one] [two]

# Script via pipeline; note the "-" to signal that the script is piped.
PSonWin> 'echo [%1] [%2]' | ins - one two
[one] [two]