Powershell: Do not require { ... } around variable names for null-conditional member access

To clarify, the milestone just means it's something the team should discuss based on the feedback, not that we are committed to making this change.

@mklement0

? should never have been allowed as part of an identifier without enclosing it in {...}.

Ummm - the initial base language for PowerShell was the Posix Shell which uses $? for the exit code q.e.d '?' is allowed in variable names unquoted.

it cannot (meaningfully) run on older versions

Sure it can:

PS>  $abc?=@{a=1; b=2; c=3}
PS>  $abc?.b                                                                                        1
PS>  2                      

Sure it can:

It cannot run meaningfully _with the new semantics_ (null-conditional access) - that's all I meant to say.

Yes, your example currently works, but _with different semantics_ (abc? _including the ?_ being considered the variable name) - but for the reasons stated this corner case is worth abandoning for the sake of making ?. work as expected.

The initial base language for PowerShell was the Posix Shell which uses $? for the exit code

$? is an _exception_ in POSIX-like shells; you can _not_ do the following:

# bash, ksh, zsh, dash (all common /bin/sh implementations)
foo?='bar' # BOOM! E.g. "bash: foo?=bar: command not found"

(And, obviously, no one's asking that $? be abolished in PowerShell.)

And, as a matter of course, it should be mentioned, that _all other_ automatic variables in the same space as $? (i.e., $^ and $$) are explicitly special-cased in the tokenizer. And, as a matter of fact, so is $?, even though it (currently) doesn't need to be.

It seems clear to me that it was expected at the time that code was written that ? would not be a valid identifier character.

I only know of one person who has used ? at the end of variable names in their scripts, and that's @StartAutomating. Since he's actually used that syntax, I thought it worth drawing his attention to this discussion.

Also @vexx32, the reason that $? is special cased in the tokenizer is because you cannot create a variable with a name that starts with ?. e.g. $??? = 'foo' will not parse. Variable names must start with alphanumeric characters or an underscore, and currently can contain alphanumeric characters, underscores, or, questionmarks. I'm in agreement that ?. should be parsed as the null-conditional member access operator, but I wanted to clarify why ? is special-cased in the tokenizer.

Thanks for drawing my attention to this chat.

For what it's worth, I have a few thoughts:

1. While these variables are rare, no one want's to play whack-a-mole when back-compat breaks their scripts
2. Deeply geeky folks aside, I'd expect that ${obj}?.Method() or $obj?.Method() will get little uptick. There's a decade of guidance out on the web of how to manage nulls out there, and I don't see that many people switching from if ($obj) { $obj.Method() } _just to get on the v7 bandwagon_
3. Additionally, most people who have problems with unexpected variable names learn the ${} syntax (you'd already hit it naturally with an underscore). I'd expect the venn diagram overlap of "people who want to use this feature" to "people who know ${} syntax already" to be near 100%
4. Since said syntax isn't as powerful as a full subexpression, and full subexpressions work both in core and not, I'd expect that most people will continue to use this going forward.

So, running the math, there are several concrete reasons _not_ to do this, and only one vague reason to do this (it _might_ be easier, for some users). Given the low prevalence of ? in variable names, you _might_ be helping more people than you harm, but in my opinion it's a coin-toss.

Given such coin-toss potential of benefit, I will paraphrase Robert McNamara "If we're darned if we do and darned if we don't, I'll pick darned if we don't".

I suggest doing it so that $obj?.Method() works as $obj?.Method() by default, and only fall back to $obj?.Method() when $obj doesn’t exist in scope, but $obj? does and #requires ‑Version isn’t v7+.

@ExE-Boss, that seems like a lot of exceptions to make for this particular rule.

I repeat my questioning of the value of this particular rule.

Past that, the question is "what else do we not know about variable syntax that we might break?"

If the consensus is to force ${...} syntax for this, support for OptionalFeatures would be appreciated here, so that folks like myself who will use this syntax (I'll be using it _a lot_) can opt out of the noise.

Since we're paraphrasing here:
"Logic clearly dictates that the needs of the many outweigh the needs of the few." - Spock

So, running the math, there are several concrete reasons _not_ to do this, and only one vague reason to do this (it _might_ be easier, for some users).

@StartAutomating: There are multiple reasons, and they are concrete, not vague. Also, one of the goals of this is code simplicity. Having to wrap variable names with {} works directly against that goal.

I'm with @mklement0 and @KirkMunro on this. Personally I find the ${...}? syntax unnatural and unnecessary complex. We have had breaking changes already and in this case it is small price to pay in exchange for clarity.

@StartAutomating

1. Additionally, most people who have problems with unexpected variable names learn the ${} syntax (you'd already hit it naturally with an underscore). I'd expect the venn diagram overlap of "people who want to use this feature" to "people who know ${} syntax already" to be near 100%

This doesn't hold up. Underscores are valid characters in variable names.

@vexx32

It seems clear to me that it was expected at the time that code was written

Nope.

Swap the chars around? $foo.?Property looks non-clashing, because property names beginning with questions marks need quoting already:

PS C:\> $foo = [pscustomobject]@{'?'=1; '?name'=2}
PS C:\> $foo.?name
At line:1 char:6
+ $foo.?name
+      ~
Missing property name after reference operator.
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : MissingPropertyName
PS C:\> $foo.'?name'
2

@HumanEquivalentUnit, unfortunately, that would invite everlasting confusion with the long-established ?. syntax used in C# and several other languages (also, you'd need to find an alternative form for the indexer (${arr}?[0]) as well, and something like $arr[0]? would invite confusion with the ternary operator)

There isn't a neat solution.

1. An unknown number of people use ? at the end of a variable names. While it is an optional feature one can say "Don't turn it on if you run those scripts" but the person turning it must check current and future scripts they get from whatever source.
2. Alternate syntax isn't a great option because it is importing something not-very-powershelly from C#
3. Changing behavior based on #requires doesn't work outside a saved script, and will fail as soon as someone copies some code from a long script without setting the tag.
   4.(dir *.ps2)?.count and (dir *.ps1)?.count show that it operand doesn't need to be a variable.
   ($Host)?.toString()/ ($null)?.toString() removes the need to put braces around the name. It's still extra keystrokes, but it is more logical and less ugy.

1. Changing behavior based on #requires doesn't work outside a saved script, and will fail as soon as someone copies some code from a long script without setting the tag.

This applies to _any_ code that uses features not supported in earlier versions.
If this were the overriding concern, no new language feature or new cmdlet / cmdlet parameter would ever be introduced.

(Hopefully, in older code you have at least Set-StrictMode -Version 1 in effect, in which case $var?.foo would fail loudly, due to no variable named var? existing).

1. removes the need to put braces around the name.

Substituting (...) for {...} is only a marginal improvement.

Whatever extra syntax is used, the problem here is the _need_ for extra syntax - for something that should just work _as-is_ - not just for typing convenience but also to meet _sensible expectations_.

1. Changing behavior based on #requires doesn't work outside a saved script, and will fail as soon as someone copies some code from a long script without setting the tag.

This applies to _any_ code that uses features not supported in earlier versions.
If this were the overriding concern, no new language feature or new cmdlet / cmdlet parameter would ever be introduced.

That wasn't quite what I meant. Further up there was suggestion that if #requires specified pwsh 7 then the processing might be different. Lighting up a new behaviour only if you put #requires isn't good .

(Hopefully, in older code you have at least Set-StrictMode -Version 1 in effect, in which case $var?.foo would fail loudly, due to no variable named var? existing).

In my experience not much uses Set-Strictmode. Because (a) there is an assumption that the default without it is "right" and (b) the frequency of using things which rely on it being off is too high. It's a common use for these operators. In any case the problem is older code where $var? does exist and does have a foo property, but now the code looks for $var ... Possibly the new processing would cause an error IF strictmode is set because $var doesn't exist....

1. removes the need to put braces around the name.

Substituting (...) for {...} is only a marginal improvement.

Yes. ($x) is slightly better than ${x} but only slightly. (Get-user "Bob")?.disable() is better than

$u = get-user "bob" 
($u)?.disable

Whatever extra syntax is used, the problem here is the _need_ for extra syntax - for something that should just work _as-is_ - not just for typing convenience but also to meet _sensible expectations_.

Yes.
$var. foo With white-space works. ${foo} ?.bar() doesn't because ? is allowed in a function or alias name (?. is legal for both). Lashing bits of C# syntax into existing syntax rules without breaking things isn't always practical Sometimes, "This is available in C#..." really deserves the answer "Yes. You know where C# is if you want it".
:-)

To maintain SemVer compatibility, this should be done in a major release (preferably 7.0.0).

Also see my suggestion above (https://github.com/PowerShell/PowerShell/issues/11379#issuecomment-566756120).

@ExE-Boss Yes, it was your comment I was referring to when I said depending on #requires might not be the good idea it sound at first.
And yes, if one is to break any existing scripts the time to do is when the major version changes, so this needs to be done quickly or wait for V8

These are still an experimental feature and are off by default in RC-1 , which I assume is going to stay the same for a shipping product. That does allow the behaviour to be changed (or even be second experimental feature) - it also should block/ warn when trying to create a variable ending with certain characters. If I enable it and then want to use scripts with errant variable names that puts the responsibility on me - of course it means that scripts I download which do that just look "broken" which is more tolerable under the "experimental" banner.

I'm not a serious PowerShell developer; I came here in search of a solution to a bug and happened to stumble upon this issue. For command line scripting, I mostly use bash. I'm probably in the demographic that Microsoft is trying to convince to switch to PowerShell Core: cross-platform developer in need of a quick scripting language that's less of a pain than bash.

I've often found many of the decisions related to PowerShell's syntax unintuitive to people who aren't primarily PowerShell developers. The last time I used PowerShell for something serious, I was appalled by the lack of a null coalescing operator, or even a ternary operator I could use instead--which resulted in me posting this popular PowerShell answer on Stack Overflow. That was in 2013, and it's easily one of the top three reasons I've mostly avoided PowerShell ever since. The whole point of a scripting language is that I want something quick and dirty with a lot of syntax sugar, not verbose if-else statements.

@StartAutomating said:

Deeply geeky folks aside, I'd expect that ${obj}?.Method() or $obj?.Method() will get little uptick. There's a decade of guidance out on the web of how to manage nulls out there, and I don't see that many people switching from if ($obj) { $obj.Method() } just to get on the v7 bandwagon

I'm just n=1, but as a non-PowerShell developer, I strongly disagree. The new ? syntax is something I would find out about pretty quickly even as someone who touches PowerShell maybe once every few months, and it would definitely increase the odds of me using PowerShell in the future in place of alternatives. If I then see that some crazy {} syntax is required for backwards compatibility, I'm going to roll my eyes and go back to whatever I was already using.

I don't understand the emphasis on backwards compatibility here. PowerShell Core already broke most of my existing scripts because they used parts of .NET that aren't in .NET Core. (I'm not complaining about that--.NET Core is great, but if you're going to break backwards compatibility, now's your chance.)

First off, I believe my initial feedback has created a kerfuffle of misunderstanding.

My supposition was that ${} would not be _that weird to PowerShell developers_ , because they would already have to know this syntax form if they had ever embedded a variable with a ? or and underscore in a string.

Thanks for the many reminders that this syntax might be awkward to non-PowerShell developers.

The second part of my feedback might have gotten lost in the mix: Because other alternatives to null-coalescing have existed for more time in PowerShell and are widely used, I don't know how much uptick the feature would have _with existing PowerShell users_. You want to come in and say "this is the one thing I need to start using PowerShell", and I can't argue with your opinion (though I might say you're missing a forest of cool features looking for a tree of language familiarity).

The third bit of my feedback seems to have been blown out of proportion. Speaking as one of the people who has occasionally used variables named like $IsThisRight? = Test-Something , my first reaction to hearing the potential news about this variable restriction was "oh well, guess I'll have to rename some variables, I wasn't getting that much for it anyways."

So, to attempt to restate and end the thread:

1. IMHO, ${} will probably not be that weird of a syntax for _existing_ PowerShell developers
2. You're welcome to think this feature is cool; to me it's a language familiarity gateway drug, and I'd highly recommend everyone reading this thread learn how to rock assigning ifs/foreaches/whiles etc.
3. As one of the impacted authors, I'm not getting something so mission critical from a ? at the end of my variables that I would be opposed to changing my code.

Or, to be pithily short:

Do this change if you want to. I have little horse in this race. If you do decide to do this, please broadcast the change clearly so that anyone else who has named variables with a ? can update accordingly.

The second part of my feedback might have gotten lost in the mix: Because other alternatives to null-coalescing have existed for more time in PowerShell and are widely used, I don't know how much uptick the feature would have with existing PowerShell users.

I do definitely agree with that, and I don't want to be dismissive of your opinion. It's always hard to make breaking changes like that, especially when the alternative might be intuitive to people who are already heavy users of the language.

You want to come in and say "this is the one thing I need to start using PowerShell", and I can't argue with your opinion (though I might say you're missing a forest of cool features looking for a tree of language familiarity).

Well, I had a feeling that most non-PowerShell developers aren't really participating in these conversations--it's pure coincidence that I happened to stumble upon this. My use of PowerShell will likely be very different from people who are writing libraries and open source tools. I figured I'd just give the perspective of someone who's using PowerShell less as a fancy framework and more to get odd tasks done as fast as possible. It's not meant to be dismissive of any other opinions--it's just another opinion from a perspective that probably isn't seen all that often here.

Better null handling isn't the one thing I need to start using PowerShell more: it's the second thing. The first was cross-platform support, and I'm very impressed with how far that's come. Personally, syntax sugar is a big deal for me when I'm choosing a language to do a given task: I want the syntax to make what I'm doing easier. If I'm writing something that needs to be maintained by a lot of people, I'm probably going to want it to be more verbose; but if I'm writing a quick script for administrative or devops tasks, I just want a quick way to test for things like null.

The third bit of my feedback seems to have been blown out of proportion. Speaking as one of the people who has occasionally used variables named like $IsThisRight? = Test-Something , my first reaction to hearing the potential news about this variable restriction was "oh well, guess I'll have to rename some variables, I wasn't getting that much for it anyways."

Favoring backwards compatibility is always a fair argument. However, in this particular case, it seems like a lot of PowerShell scripts are on the verge of breaking due to the transition to Core anyway. When backwards compatibility breaks on a regular basis, it's frustrating, but I would be in favor of any breaking changes happening all at once, which means this is probably a good time.

In a scenario in which you're trying to target a whole new demographic--that is, non-Windows-centric developers--it might be necessary to make some breaking changes to compete with what's already out there.

If you do decide to do this, please broadcast the change clearly so that anyone else who has named variables with a ? can update accordingly.

To be clear, I'm fine with needing to opt into the feature--that wouldn't deter me from using PowerShell. I'm already accustomed to needing a bunch of set commands at the top of my bash scripts. I wouldn't consider it ideal, but it'd have little impact on my decision to use PowerShell as long as it's well-documented.

Personally, syntax sugar is a big deal for me when I'm choosing a language to do a given task

Then you've come to the right language! PowerShell is positively sacchrine with syntactic sugar.

To educate those who don't know, you can "null coalese" several ways in PowerShell:
~
$ListWhereNotNull = $null, 1, 2, $null, 3 -ne $null # This will check that each item is not null, and return a list of 1,2,3
$FirstNonNull = @($null, 1, $null,2 -ne $null)[0] # This will return the first non-null
$lastNonNull = @($null, 1, 2, $null -ne $null)[-1] # This will return the last non-null, using a negative index
$TheObjectPipelineWay = $null, 1, $null, 2 | Where-Object { $_ } | Select-Object -First 1
$TheAssigningForeachWay = foreach ($item in $null, 1, 2, $null, 3, 4) {
if ($item -ne $null) { $item; break }
}
$FirstViaMultipleAssignment, $RestIDontCareAbout = $null, 1, $null, 2, 3,4, $null -ne $null
~
That's enough for now, but it should prove the point on the language syntax having much flexibility. All of those demos will work all the way back to PowerShell 1.0

However, in this particular case, it seems like a lot of PowerShell scripts are on the verge of breaking due to the transition to Core anyway.

In this you are incorrect. Core actually has very few syntax breaking changes, and more modules than you'd think work just fine on core (+- some issues with Linux vs Windows pathing/newlines). In more practical terms, your module is less likely to work on Core if you have some sort of Windows platform specific API dependency (e.g. I'm not going to hold my breath to get my WPF wrappers working on Linux).

I still am fine either way with the syntax change: I just felt that two of the assertions in the latest reply needed to be addressed and called out.

Re: Syntactic sugar, my sarcastic mind quotes the movie "Snatch": "I'm already sweet enough"
Re: breaking change potentials in Core: We have few now, and let's keep it that way as much as we can.

@StartAutomating : Are you saying i can't shoot? because there's still place for more sugar I think 😜

To educate those who don't know, you can "null coalese" several ways in PowerShell:

I did figure that out (that's my own answer that I wrote), but it felt like a kludge. It's a cool idea with a lot of potential, but as one commenter on that Stack Overflow answer pointed out, evaluation doesn't short-circuit. The ?. operator means that this is usually no longer an issue when chaining calls/properties. Back in 2013, though, it wasn't available--and, even then, there's an awful lot going on in these examples, and you can see in the comments how people found that confusing. I had to break down exactly what these examples do symbol-by-symbol.

In this you are incorrect. Core actually has very few syntax breaking changes, and more modules than you'd think work just fine on core (+- some issues with Linux vs Windows pathing/newlines). In more practical terms, your module is less likely to work on Core if you have some sort of Windows platform specific API dependency (e.g. I'm not going to hold my breath to get my WPF wrappers working on Linux).

I'm just speaking from experience in this regard, rather than actually going through and gathering proper statistics. For example, pre-Core, I could generate a password with [System.Web.Security.Membership]::GeneratePassword(32, 6), but System.Web isn't available in Core. (I know I could probably load the old System.Web.dll, but I'd rather not do that if I don't have to.) Syntax changes are actually easier for me to resolve than missing APIs. Generating a bunch of Docker secrets for a devkit seems like a great place to use a quick script, but it's surprisingly more verbose in PowerShell Core than bash. Of course, bash's solutions often look pretty ugly.

Re: Syntactic sugar, my sarcastic mind quotes the movie "Snatch": "I'm already sweet enough"

It depends. Sometimes I feel like PowerShell is extraordinarily pleasant. Dealing with null is one of those key areas where I usually don't feel that way--although I'd say the same for a lot of languages. I also find the traditional inline PowerShell if-else statement to be cumbersome compared to a ternary operator or bash's && and || chaining. Since this is one of the key issues that caused me to drift away from PowerShell years ago, I figured it was worth mentioning.

On the other hand, I deal with a wide variety of languages on a daily basis, and no matter what I'm writing, I'm always thinking, "this would be so much easier in language X!"--and then when I'm using language X, I'm thinking, "this would be so much easier in language Y!" But hey, if I get an opportunity to say that out loud and maybe sway the direction of the language, I'll take it.

I'm not really interested in changing any minds here, just adding another perspective.

@StartAutomating : Are you saying i can't shoot? because there's still place for more sugar I think 😜

I always love me some new syntax sugar! I select programming languages like a kid going trick-or-treating on Halloween.

@Zenexer : I really like Powershell, I do. I'm fortunate enough to do PS extensively and daily at work. I love the flexibility, especially the symbiosis with .NET, oh man! That's why I care about its direction and that's why the ${...}? syntax is so hard to accept to me.

Good points, but let me clarify a few things.

For better or worse, PowerShell to date hasn't used semantic versioning.
A new major version only brought _new_ features, with an ironclad commitment to backward compatibility.
Breaking changes started to creep in in Core, in part of necessity due to going cross-platform.
Major version v7.0 was chosen to signal an _increase_ in backward-compatibility: an attempt to provide a viable replacement for Windows PowerShell.

While many historical problems cannot be fixed without seriously breaking existing code and can therefore only be offered as optional (opt-in) features,
bucket 3 breaking changes should always be an option: make a fix / improvement that has the _potential_ to break existing code, but is unlikely to in practice; on balance, it is worth making to better the language.

As argued in the OP, the case at hand strikes me as a bucket 3 change; while we know that _some_ scripts will break (@StartAutomating will know how to fix his scripts), the data suggests that such cases are quite rare.

Why is it rare?

Because using ? in variable names doesn't occur to most people altogether, and if it does, they shouldn't expect it to work _without {...}_: If all of the following work only with {...}, why would expect it to be different for ??: . , ; (: can't be used at all, because it is always interpreted as a drive separator)

Allowing names such as $var? or $va?r without also requiring {...} was a permissiveness that asked for trouble, and the trouble has arrived: it gets in the way of the evolution of the language, as in this case.

There are 3 new related features (at least related in syntactic form) inspired by C#:

• Ternary conditionals - will be a bona fide feature in 7.0
  
  
  
  • (Get-Date).Second % 2 ? 'odd' : 'even'
  
  
• Null-coalescing operator - will be a bona fide feature in 7.0
  
  
  
  • $var ?? 'default' and $var ??= 'default'
  
  
• Null-conditional operator - the issue at hand - will be an _experimental_ feature in 7.0
  
  
  
  • $possiblyNull?.ToString()
  
  

All of them have ? as part of their syntax, and all of them are affected by the burden of legacy $var? parsing, though ?. most obviously so:

• While you could argue that something like $var?1:0 is only of interest to code golfers, it would work just fine if ? had syntactic function - currently you must use a _space_ before the ?

• By contrast, it strikes me as reasonable for someone to attempt $var??='bar' (perhaps less so $baz = $foo??$bar), which currently also require a space.

That is, if we stick with the current parsing of $var?, we're encumbering _three_ new features (albeit to varying degrees), 2 of which are already generally available (not experimental).

The issue with requiring {...} is _not_ just the inconvenience of typing: it is just as much about _not expecting the need for it_ in the first place, and later _forgetting the need for it_ (with potentially subtle malfunctioning) - because it so counter-intuitive.

@ExE-Boss, I agree with @jhoneill that implementing version-conditional behavior would be problematic. I don't think there is need to encumber the engine with backward-compatibility special-casing in this case.

@mklement0 : It is unnatural/counter-intuitive/you-name-it. Brackets are hard, curly brackets even harder. There's no need for them in this case.

@mklement0 since agreement seems to be breaking out ... I think that's a good summary.

I'm yet to be completely won over by adding C# constructs to PowerShell. We already have issues in many places that there just aren't enough symbols on the keyboard. Is ">" "Greater than", or "redirect to" ? Is "=" assignment or test-equality (different languages use := for assign or == for equality). People jumping between languages will forget to use -eq or -gtand we're stuck with that... Is + arithmetic addition, or string/array concatenation. * is both multiplication and a wild card. % is Modulo and an alias forforEach-object.

Poor ? Is the alias forWhere-Object. And is the name of an automatic variable. But try doing Get-Alias ? orGet-Variable ? ... and then it does the job of a wildcard.

Do Get-PSdrive and you see "variable" etc don't have a ":" on the end of the name, adding one : onto a name causes it to be treated as a drive. Unless of course you refer have written "$drive="C" and follow it with "$drive:temp" Anyone new to PowerShell (and the some old hands, when they're not thinking about it) will be baffled to know they've just invented a named scope "drive".

So against that ... one might say 'Create a new syntax with further uses for ":" and "?" are you crazy?'
IIRC, on twitter @BrucePay described the ternary operator as "Not very PowerShelly" and I tend to agree. But the "Make it more like C#" voices won the day...

Most people would write $PSEdition.length -gt 2
but $PSEdition. length-gt2 also works (you can even have a line break between the two halves.) "." is also overloaded serving as "current directory", "run in this scope" and "member" However in PS V1-V5 I can't think of any operators which require a leading space; "." is odd in breaking if one _is_ present. But, in order to parse, the ternary operator _might_ need a leading space
$host.debuggerEnabled?0:1 is OK because "?" assumed _not_ to part of a property
$IsCoreCLR?0:1 isn't because "?" is assumed to be part of the variable name.

The same is true for the null coalescing operators
$null?? "default" needs a space $true.length?? "default" doesn't.

Put the original use case at the top of the issue to one side for a moment. The above cases show that something isn't quite working. It would be fixed if variable names containing "?" needed to be braced like names containing "." or "+" or ":" , and @StartAutomating who is a main user of "?" in names thinks that is a _break which could be handled_. Before we even get to null-members, it sounds like 'find a way to do it already'

I think there is a short time window to put in an experimental feature requiring ? to be braced in names, and if that feature is ON - which should be the default - all the new ? operators should parse accordingly. Only if that feature is OFF would the parser require spaces or braces with these operators.. That is a breaking change, some scripts (a small unquantifiable number) would be broken unless an option was set (and disallowing the variable name should prevent $IsIt?.tostring() being the first error in a script, so the scripts should fail, not run dangerously). and ideally that should not happen with a point release (you're correct about PowerShell not really using sem-ver, but it's still a good principle)

The issue with requiring {...} is not just the inconvenience of typing: it is just as much about not expecting the need for it in the first place, and later forgetting the need for it (with potentially subtle malfunctioning) - because it so counter-intuitive.

This is a good point and is largely what would annoy me about ${...?} in practice. Even if I know to use {}, there's a strong chance that I'll eventually forget, and that might result in subtle bugs that aren't obvious at first, since it'll probably still be syntactically valid. $var??='bar' is a good example of that, although I'm normally pretty careful about spacing.

Thanks, @jhoneill, that's a nice illustration of how many symbols do a lot of double (triple, ...) duty in PowerShell.
For the most part, once you are aware of all uses, that's not a problem, for the following reasons:

• different contexts (e.g., * as an operator vs. * as a wildcard char.)
  
  
  
  • The different contexts for ? actually have something in common, at a certain level of abstraction: You can conceive of the ? in Get-ChildItem | ? Target, $var?.Target, $?, $var ?? 'none' as all involving a kind of _question_ and therefore _test_ or _conditional behavior_.
  
  
• sensible polymorphic behavior (e.g., + performing concatenation with string).

A problematic example is & (call operator vs. background operator), which is used in the same context with different semantics, depending only on its _position_.

It would be fixed if variable names containing "?" needed to be braced like names containing "." or "+" or ":"

Indeed, and that bears repeating: the ability to use ? in variable names won't go away - but you'll have to use {...} going forward: ${var?} = @{ foo=1 }; ${var?}?.foo

As you demonstrate, from the perspective of someone sensibly _not_ expecting ? to be a valid char. in an _unbraced_ variable reference, ?., ?? and ? : have effectively made the language (situationally) whitespace-sensitive for these operators, which I know the team wants to avoid - for good reasons, because PowerShell generally isn't whitespace-sensitive (at least not one a single line, . for dot-sourcing being the justifiable exception; ? as the Where-Object alias requiring a space after may be surprising (e.g., gci|?target doesn't work), but command names _do_ need to be separated from their arguments with whitespace).

Being able to put whitespace between . and a member name ($obj. foo) primarily makes sense in multi-line statements, though the arguably more readable form - familiar from other languages - is to (also) allow whitespace _before_ the .:

# This does NOT work - the . must be *immediately after* the expression / member
(Get-Date)
  .ToUniversalTime()  
  .TimeOfDay

Being able to do so would mirror the recently added ability to place | on the _next_ line:

 # Already works in PS Core
Get-Date
  | % ToUniversalTime
  | % TimeOfDay

However, this would introduce unresolvable ambiguities with . as the dot-sourcing operator and command names that start with . - however unlikely. (e.g., . foo on a new line could be a property access or a command that dot-sources a function or script named foo (which would have to be in $env:PATH); .foo could be the name of a _command_).

@KirkMunro's multi line continuation RFC, while focused on allowing _commands_ to span multiple lines, would give us the next best solution - see https://github.com/PowerShell/PowerShell-RFC/pull/179#issuecomment-498734875.

As for the experimental feature being on by default: Fortunately, _preview_ releases - but not release candidates - now turn _all_ experimental features on by default; however, note that this is persistently overridden if you've ever run Enable-ExperimentalFeature either _without_ -Scope or with -Scope CurrentUser with whatever features you enabled there.

@mklement0 While I agree with basically all of that, I fear we might be headed for derailing this particular discussion a little bit with the extra bits and pieces you have there; if you want to address the line-continuation around the dot-property operator, you may want to open a new issue for that if there isn't one already. 🙂

I agree that continuing to allow ? as a normal variable name character is at best extremely confusing for users, as the character's use suddenly makes things whitespace sensitive, and historically at least, I don't think variable names have _ever_ been whitespace-sensitive in the past.

Introducing that ambiguity is, I think, more of a problem than making a breaking change and disallowing ? from use in a variable name. With the breaking change, we can easily build a PSSA rule to alert authors that the syntax is no longer permitted. However, the reverse is not so easily true, and unlike the breaking change it will not necessarily error out; it can very easily end up with users accidentally referencing the wrong variable silently, without knowing that's what they're doing.

@PowerShell/powershell-committee discussed this, we had analyzed the PowerShell corpus of scripts and there was quite a bit of usage of variable names ending with the question mark. As such, we cannot break those folks and braces around PowerShell variable names is an existing language feature to distinguish the variable name from other characters (like within a string).

we had analyzed the PowerShell corpus of scripts and there was quite a bit of usage of variable names ending with the question mark

Thanks, @SteveL-MSFT - can you please share the results of this analysis?

Yeah I'd be curious to see some of that, because I'm aware some community members did their own analysis and didn't get anything close to that kind of result.

I would also like to see the results and method used. I did my own analysis of the PowerShell Corpus last night using the AST which took almost 6 hours to complete on my machine. There were a small number of files not able to be parsed due to Windows Defender quarantining the file.

I found 11 unique variable names across 8 files ending in a question mark (excluding $? which showed up 8846 times). There were a total of 1,895,983, unique variables across 408,423 files. That means 0.0006% of all unique PowerShell variables in the corpus has a variable name ending with a question mark.

Here are the 8 files and 11 unique variables:

File                 : C:\Temp\PowerShellCorpus\Github\alanrenouf_PowerActions\Kill-VM.ps1
QuestionMarkVars     : vParam?
QuestionMarkVarCount : 1

File                 : C:\Temp\PowerShellCorpus\Github\anthonywlee_PowerShell\Send_Notification.ps1
QuestionMarkVars     : To?
QuestionMarkVarCount : 1

File                 : C:\Temp\PowerShellCorpus\Github\aspear_My-PowerCLI-scripts\my-labotomize-vms.ps1
QuestionMarkVars     : vParam?
QuestionMarkVarCount : 1

File                 : C:\Temp\PowerShellCorpus\Github\jlundstrom_Scripts\Powershell\7Zip SFX Creator\Build.ps1
QuestionMarkVars     : Title?
QuestionMarkVarCount : 1

File                 : C:\Temp\PowerShellCorpus\Github\pslaughter_Working\Setup-Host.ps1
QuestionMarkVars     : HostIp?
QuestionMarkVarCount : 1

File                 : C:\Temp\PowerShellCorpus\Github\stevenayers_PowerShell-Scripts\Random Functions\Add-OutlookConferenceRegionNumber.ps1
QuestionMarkVars     : {key1?, key2?, key3?, key4?}
QuestionMarkVarCount : 4

File                 : C:\Temp\PowerShellCorpus\Github\unixboy_powershell-stufff\stopvm.ps1
QuestionMarkVars     : vParam?
QuestionMarkVarCount : 1

File                 : C:\Temp\PowerShellCorpus\PowerShellGallery\PsHg\0.6.2\PsHg.psm1
QuestionMarkVars     : PsHg?
QuestionMarkVarCount : 1

This is the script used to generate the report:

Get-ChildItem -Path C:\Temp\PowerShellCorpus -Include *.ps1, *.psm1 -File -Recurse | ForEach-Object -Parallel {
    try {
        $content = $PSItem | Get-Content

        $ast = [System.Management.Automation.Language.Parser]::ParseInput($content, [ref]$null, [ref]$null)
        $variables = $ast.FindAll({$args[0] -is [System.Management.Automation.Language.VariableExpressionAst ]}, $true)

        # First query because I forgot to exclude $? variables
        # $nonQuestionMark = $variables | Where-Object VariablePath -notmatch '\?$' | Select-Object -Unique
        # $QuestionMark = $variables | Where-Object VariablePath -match '\?$' | Select-Object -Unique

        $nonQuestionMark = $variables |
        Where-Object VariablePath -notmatch '\?$' |
        ForEach-Object { $_.VariablePath.UserPath.ToString() } |
        Select-Object -Unique

        $QuestionMark = $variables |
        Where-Object { $_.VariablePath -match '\?$' -and $_.VariablePath.UserPath -ne '?' } |
        ForEach-Object { $_.VariablePath.UserPath.ToString() } |
        Select-Object -Unique

        $output = [pscustomobject]@{
            File = $PSItem
            QuestionMarkVars = $QuestionMark
            QuestionMarkVarCount = $QuestionMark.Count
            NonQuestionMarkVars = $nonQuestionMark
            NonQuestionMarkVarCount = $nonQuestionMark.Count
        }

        $output
    }
    catch {
        throw
    }
}

Let me know if there is any issue with the method used to generate this report. Here is an example script used to test the audit.

$abc = 'test'
$isAwesome? = $true
$?

$test = {
    $?
    $isabc? = { $adce? = 'test' }
    ${def?} = 123
    ${is a bad var name?} = $isabc?
}

Results in:

File                    : C:\temp\variable.ps1
QuestionMarkVars        : {isAwesome?, isabc?, adce?, def?, is a bad var name?}
QuestionMarkVarCount    : 5
NonQuestionMarkVars     : {abc, true, test}
NonQuestionMarkVarCount : 3

As they say in math class, "please show your work" 😄

Excellent sleuthing, @ThomasNieto - thank you.

I think you can potentially eliminate even more cases, given that the form ${...} will continue to work with the proposed change - e.g., it's only $isAwesome? we need to worry about, not ${isAwesome?}

With your example file:

$variables.Extent.Text.Where({ $_ -match '(?<!\$)\?$' }) | Select-Object -Unique

we only get:

$isAwesome?
$isabc?
$adce?

That is, ${def?} and ${is a bad var name?} needn't be considered.

P.S.: It's probably also better to use $PSItem | Get-Content -Raw to read the entire script as a single string.

@mklement0 That's correct, I added those cases right before posting to ensure the script was not excluding variables with the curly brace notation all together.

After updating the script with your recommendations the test file produces the following output like we expect.

File                    : C:\temp\variable.ps1
QuestionMarkVars        : {isAwesome?, isabc?, adce?}
QuestionMarkVarCount    : 3
NonQuestionMarkVars     : {abc, true, test}
NonQuestionMarkVarCount : 3

I ran the updated script on those 8 files and none of them are using the curly brace notation so the result is still 11 unique variables would be impacted with the proposed change.

Updated script:

Get-ChildItem -Path C:\Temp\PowerShellCorpus -Include *.ps1, *.psm1 -File -Recurse | ForEach-Object -Parallel {
    try {
        $content = $PSItem | Get-Content -Raw

        $ast = [System.Management.Automation.Language.Parser]::ParseInput($content, [ref]$null, [ref]$null)
        $variables = $ast.FindAll({$args[0] -is [System.Management.Automation.Language.VariableExpressionAst ]}, $true)

        $nonQuestionMark = $variables |
        Where-Object VariablePath -notmatch '\?$' |
        ForEach-Object { $_.VariablePath.UserPath.ToString() } |
        Select-Object -Unique

        $QuestionMark = $variables |
        Where-Object { $_.VariablePath -match '\?$' -and $_.VariablePath.UserPath -ne '?' -and $_.Extent.Text -match '(?<!\$)\?$' } |
        ForEach-Object { $_.VariablePath.UserPath.ToString() } |
        Select-Object -Unique

        $output = [pscustomobject]@{
            File = $PSItem
            QuestionMarkVars = $QuestionMark
            QuestionMarkVarCount = $QuestionMark.Count
            NonQuestionMarkVars = $nonQuestionMark
            NonQuestionMarkVarCount = $nonQuestionMark.Count
        }

        $output
    }
    catch {
        throw
    }
}

Thanks, @ThomasNieto - I was hoping we could get the percentage down to 0.0005%... (just kidding).

So the committee must have used a different - private? - corpus.

So, in addition to deciding the issue at hand, the question arises: does the publicly available corpus need updating? Is it officially maintained (currently dated 26 July 2017)?

@SteveL-MSFT, can you please clarify?

@mklement0 I rounded up so if you want to feel just a bit better its actually 0.00058% 😄

In https://github.com/PowerShell/PowerShell-RFC/pull/223#discussion_r318340339 @adityapatwardhan did an analysis using the same public corpus I used and came up with 62% with variables ending with a question mark. I didn't see a response on how he came up with that percentage. Did the PS committee use that analysis to determine this request?

Looking at the linked comment, I think what @adityapatwardhan is saying that _among all variables that have ? in their name_ 62% have a ? as the name's _last_ character.

By contrast, your analysis of the actual prevalence of (curly-brace-less) ending-in-? variables (as a percentage of variable usage overall) seems much more relevant.

At this point, it sounds like the committee is just saying "we don't want to" and if that's the case, why even have this feature if it's going to be clunky and un-used as a result?

@ThomasNieto likewise I did an analysis somewhere around these issues and came up to similar results to yours.

@mburszley Thanks for doing your analysis. I wanted to use another method (ast vs regex) to confirm the results you got last year.

I hear you, @mburszley, but re:

the committee is just saying "we don't want to"

If this truly was a _fiat_ decision - one not grounded in analysis of real-world usage - it wouldn't bode well for the community at large, especially given that the _stated_ reason was a grounding in such analysis.

Thus, we should give the committee a chance to show the real-world usage analysis the decision was based on - or to recognize that the analysis fell short and reconsider the decision.

What about my suggestion in https://github.com/PowerShell/PowerShell/issues/11379#issuecomment-566756120?

I suggest doing it so that $obj?.Method() works as $obj?.Method() by default, and only fall back to $obj?.Method() when $obj doesn’t exist in scope, but $obj? does and #requires ‑Version isn’t v7+.

I would expect that it would address most of the backwards compatibility concerns.

@ThomasNieto

@mklement0 is correct, I did mean that among the variables using ?, 62% use it at the end.

From the results shown in comment here seems all of the variables which use ? use it at the end. Hence the proposal to make a non-breaking change.

If the requirement for {..} is removed then the script will still parse but the semantics will be different, which in my opinion is very dangerous.

Sure... but analysis shows literally almost nobody is using them.

Same as any other breaking change, it can be announced, introduced as an experimental feature for a time, and then documented and brought into stable.

I see no reason to recommend an esoteric syntax when ? as an identifier character is not actually being used in a remotely significant number of cases.

If the requirement for {..} is removed then the script will still parse but the semantics will be different, which in my opinion is very dangerous.

Aren't you also worried about folks using ?. the way they do in C/C++/C# and getting the wrong result with no indication they did anything wrong? For example:

PS> $res = "My length is 15"
PS> $res?.Length
0

Sure, strict-mode will find this but I don't know that a lot of folks use strict mode. So, to prevent a very small percentage of scripts from breaking, it seems we're setting up a lot of folks for failure using ?.. I'm not crazy about the trade-off. That said, I definitely understand the desire to avoid breaking folks even if it's a very small number of folks.

Maybe PowerShell needs an equivalent of Enable-ExperimentalFeature call it Enable-PSFeature, for features that are no longer in the experimental state but are not ready to be turned on by default. For scripts that execute with code like this $?.GetType() you could start emitting warnings that in the future this will break and that you should use ${?}.GetType() instead. Individual scripts could enable the feature with a #requires -version 7.1 -feature PSNullConditionalOperators. Then maybe whenever PS 8 hits, the feature could be enabled by default.

@rkeithhill 's example goes to straight to the point.
Forget for a moment what you know about how PowerShell works today. When you see

PS> $res?.Length

It looks like _name, operator, name_ - which of course it is.
Which would make more sense: _"?" is part of the operator_ , or _"?" is part of the first name-element_ ?
If these operators are needed (and I think they are "not PowerShelly") its a choice between something which will be used incorrectly and a small amount of breakage.

I would start putting rules into psScriptAnalyzer _now_ to say it's bad style to use certain [legal] characters in variable names.

I don't like Enable-feature. Someone puts it in script to make something they want to use work, and an old script breaks if run after the new script, (but works the rest of the time). I've seen _third party script X fails after running script Y_ because Y turns on strict mode and X relies on it being off; the authors of both scripts are in my bad books.

I would start putting rules into psScriptAnalyzer now to say it's bad style to use certain [legal] characters in variable names.

Here's the problem: if you change the way PowerShell tokenises variables, PSScriptAnalyzer will see them differently too. If you have a variable like $res?, then looking at something like this:

{
$res?.Length
}.Ast.EndBlock.Statements[0].PipelineElements[0].Expression.Expression.VariablePath.UserPath

gives you

res?

BUT, if you change the way PowerShell is parsed in, say, 7.2, then you will get

Now PSScriptAnalyzer can't see the ?, because it's no longer part of the variable name (by definition, since it's the change we're trying to test for). This is because PSScriptAnalyzer is a PowerShell module and reuses the same parser that ships in PowerShell. (And it would be totally unmaintainable and a recipe for disaster if it tried to reimplement it).

There are ways around this, like conditionally compiling PSScriptAnalyzer (and shipping a whole new assembly) to test the new 7.2 logic to supply the same diagnostic. But then you've increased complexity everywhere (or alternatively decreased your test surface) to service a somewhat cosmetic scenario.

Worse, all the other tools out there that also rely on the AST won't have the capacity to sustain this kind of complexity even if PSScriptAnalyzer does. Most people building AST-targeting tools out there aren't going to compile against individual minor versions of PowerShell for perfect correctness. They probably won't even know about such a small breaking change, especially since it's not an error in any version; it's the most insidious form of breaking change, where one behaviour quietly becomes another.

That's basically what makes parser/tokeniser-level changes like this one more dangerous. Even PSScriptAnalyzer must bend over backwards to service them correctly, meaning they must be weighed against such onerous changes.

@rjmholt, while I think we can all appreciate how tricky a change like this is _in principle_, in this particular case it doesn't matter _in practice_, and I don't think a PSSA rule is even needed:

• @ThomasNieto's analysis has shown that - if we believe the corpus to be representative of all PowerShell code out there - that _virtually no one uses variables with names that end in ?_.

• If I were to guess, most people wouldn't even _think_ to use such variable names, because they wouldn't expect them to work (me included), and probably don't even notice the exception that the automatic $? variable constitutes (whose name is equally an exception in POSIX-compatible shells such as bash).

  • In fact, looking closer at @ThomasNieto's analysis reveals that, among the 8 files listed:
    
    
    
    • 5 contain only _false positives_, namely variables uses inside _strings_ such as "Are you sure you want to kill $vParam?" - in fact, these uses are _broken_, and reveal that the script authors did _not_ expect ? to be part of the variable name.
    
    
    • 2 of those files are no longer publicly available.
    
    
    • Only _1_ of them is a bona fide use of ?-ending variables - as _Boolean_ variables (e.g., $key1?), which, as an aside, are therefore _not_ combined with the member-access operator, . /cc @stevenayers.
    
    

Based on the above, this strikes me as a textbook Bucket 3: Unlikely Grey Area change, which is therefore _acceptable_ - and I think the _benefits_ of this change have been argued persuasively.

_Documenting_ the change in behavior (with a rationale) should suffice.

Of course:

1. this is predicated on (a) the analysis being correct and (b) the publicly available corpus being representative.

2. it is starkly at odds with the committee's claim that "there was quite a bit of usage of variable names ending with the question mark" (which, of course, would only be a problem if such variable names aren't enclosed in {...}).

Proceeding on the assumption that 1. is true (we haven't heard anything to support 2.), we have two options:

• Rip off the band-aid and simply disallow ? in variable names going forward, unless enclosed in {...} (with the obvious exception of $?) - this will break the vanishingly small proportion of scripts that currently rely on that.

  • That is, something like $key1? that is neither followed by . nor another ? ($key1??'else') nor a ternary expression ($key1?1:0) would cause a _parser error_.
    
    
    
    • As the null-coalescing and ternary operator examples show, they too would benefit from this change - currently, you either need a _space_ - $key1 ?1:0 / $key1 ??'else' - or {...} - ${key1}?1:0 / ${key1}??'else'
    
    
  • Inside _expandable strings_, ? would then no longer be considered part of a (non-{...}-enclosed) variable name, so we would actually _fix_ existing scripts that have mistakenly used strings such as "Are you sure you want to kill $vParam?". 😁

• If we really feel we need to avoid breaking the vanishingly small proportion of existing scripts, we _could_ consider the logic proposed by @ExE-Boss, but I don't think we want to introduce this conceptual complexity - let alone implementation complexity (which I can't really speak to).

@SteveL-MSFT

Thanks for discussing this issue in the September community call (https://aka.ms/PSCommunityCall; as of this writing, the September call's recording has not been posted there, but is accessible at https://aka.ms/JoinPSCall, until the next call), starting at 46:00, based on @ThomasNieto's suggestion in the prior solicitation for discussion topics at https://github.com/PowerShell/PowerShell-RFC/issues/260:

(I was hoping to provide a direct link to the relevant portion of the recording, but that will have to wait until it is posted to YouTube and linked to at https://aka.ms/PSCommunityCall.)

Based on your remarks in the recording:

• Given that you indicated willingness to revisit this, please reopen this issue and re-tag it as Review - Committee.

• As for your remarks that enclosing variable names in curly braces is a preexisting feature and that people object to having to use them in this case on _aesthetic_ grounds:

  • I don't think anyone here _isn't_ aware that {...} _can_ and sometimes _must_ be used to disambiguate variable names.
  • It is also not about _aesthetics_, it is primarily about _not getting the expected behavior_ in manner that _fails quietly_, due to _not expecting to have to use {...} in this case_; a secondary concern is typing inconvenience.

GitHub

PowerShell/PowerShell-RFC

RFC (Request for Comments) documents for community feedback on design changes and improvements to PowerShell ecosystem - PowerShell/PowerShell-RFC

Microsoft Teams

Join conversation

I'm concerned that this issue - now seemingly indefinitely in a closed state - will fall through the cracks, so I've opened #14025 as a reminder to revisit it, and I invite everyone who's interested to show their support there.