Julia: allow overloading of a.b field access syntax

The ability to use dots as syntactic sugar for mutator/accessor methods would be nice for lots of things. I've always appreciated this in languages that provide it, so that you can turn structure fields into more complicated abstractions without breaking the API.

+1

I have an absolutely awesome way to implement this.

Interested in talking about it? I know that Tom Short is really interested in having this for DataFrames, although I've come to be increasingly skeptical about the wisdom of using this feature.

This would make calling Python code (via PyCall) significantly nicer, since currently I'm forced to do a[:b] instead of a.b.

@JeffBezanson, any chance of having this for 0.3? Would be great for inter-language interop, both for PyCall and for JavaCall (cc @aviks).

@JeffBezanson if _not_, is there any chance you could give some direction on how you want this implemented? (I have an absolutely awesome way to implement this.)

In my experience, there is no faster nor surer way to get Jeff to implement something than to implement a version of it that he doesn't like ;-)

The basic idea is that you implement

getfield(x::MyType, ::Field{:name}) = ...

so that you can overload it per-field. That allows access to "real" fields to keep working transparently. With suitable fallbacks getfield(::MyType, ::Symbol) also works.

The biggest issue is that modules have special behavior with respect to .. In theory, this would just be another method of getfield, but the problem is that we need to resolve module references _earlier_ since they basically behave like global variables. I think we will have to keep this a special case in the behavior of .. There is also a bit of a compiler efficiency concern, due to analyzing (# types) * (# fields) extra function definitions. But for that we will just see what happens.

@JeffBezanson Do you also refer to const behavior in modules? It would be useful to have a user type emulating a module and be able to tell the compiler when the result of a dynamic field lookup is infact constant. (another approach would be to start with an actual module and be able to "trap" a failed jl_get_global and inject new bindings on demand)

I would find that to be very useful in combination with #5395. Then one be able to intercept a call to a undefined function or method MyMod.newfunction(new signature) and generate bindings to a (possibly large) API on demand. This would then be cached as usual const bindings I guess.

Let me, a simple Julia newbie, present a little concern: I think the possibility to overload the dot operator might imply that field access "purity" is somehow lost.

The user would generally lose the knowledge if doing a.b is just an access to a reference/value or if there can be a huge function machinery being called behind. I'm not sure how that could be bad though, it is just a feeling...

On the other hand, I see that indeed this is a big wish for syntax sugar for many cases (PyCall, Dataframes...), which is perfectly understandable.
Maybe it is time for .. #2614?

I support doing this.

But the purity does have something to say for it, even if one can use names(Foo) to figure out what the real components of Foo are.

The purity argument is closely related to the main practical concern I have, which is how one handles name conflicts when the fields of the type interfere with names you might hope to use. In DataFrames, I think we'd resolve this by banning the use of columns and colindex as column names, but wanted to know what people's plan was for this.

I guess getfield(x::MyType, ::Field{:foo}) = ... would have to be forbidden when MyType has a field foo, otherwise the access to the real field would be lost (or a way to force access to the field would have to be available).
But then getfield could only be defined for concrete types, since abstract ones know nothing about fields.

(Meanwhile, I stumbled upon this about C++.)

It's not a major problem. We can provide something like Core.getfield(x, :f) to force access to the real fields.

Ok, maybe I'm sold. But then defining a shortcut to Core.getfield(x, :f) (e.g., x..f) will be nice, otherwise internal code of types overloading the . for all symbols (dataframes, probably dictionaries) have to be crowded with Core.getfields.

I'm not worried about the purity aspect - until we have this, the only code
that should be using field access at all is code that belongs to the
implementation of a given type. When field access is part of an api, you
have to document it, as with any api. I agree that it might be handy with
some shortcut syntax for core.getfield though, when writing those
implementations.

It had already been pointed out in #4935, but let's pull it to here: dot overloading can overlap a little with classical Julian multiple dispatch if not properly used, since we can start doing

getfield(x::MyType, ::Field{:size}) = .........
for i=1:y.size .....

instead of

size(x::MyType) = ..........
for i=1:size(y) ....

While the dot would be great to access items in collections (Dataframes, Dicts, PyObjects), it can somehow change the way object properties (not fields) are accessed.

I think one thing to consider is that if you can overload accessing field, you should also be able to overload _setting_ a field. Else this will be inconsistent and frustrating. Are you OK to go that far?

@nalimilan, one absolutely needs a setfield! in addition to getfield. (Similar to setindex! vs. getindex for []). I don't think this is controversial.

Agree with @stevengj: DataFrames will definitely be implementing setfield! for columns.

I support this.

Experience with other languages (e.g. C# and Python) does show that the dot syntax does have a lot of practical value. The way that it is implemented through specialized methods largely addresses the concern of performance regression.

It is, however, important to ensure that the _inlineability_ of a method won't be seriously affected by this change. For example, something like f(x) = g(x.a) + h(x.b) won't become suddenly un-inlineable after this lands.

If we decide to make this happen, it is useful to also provide macros to make the definition of property easier, which might look like:

# let A be a type, and foo a property name
@property (a::A).foo = begin
    # compute the return the property value
end

# for simpler cases, this can be simplified to
@property (a::A).foo2 = (2 * a.foo)

# set property 
@setproperty (a::A).foo v::V begin
    # codes for setting value v to a property a.foo
end

Behind the scene, all these can be translated to the method definitions.

I'm not convinced that @property (a::A).foo = is all that much easier than getproperty(a::A, ::Field{foo}) = ...

In any case, better syntactic sugar is something that can be added after the basic functionality lands.

Regarding inlining, as long the field access is inlined before the decision is made whether to inline the surrounding function, then I don't see why it would be impacted. But maybe this is not the order in which inlining is currently done?

getproperty(a::A, ::Field{:foo}) = strikes me as there are too many colons :-) I agree that this is a minor thing, and probably we don't need to worry about that right now.

My concern is whether this would cause performance regression. I am not very clear about the internal code generation mechanism. @JeffBezanson may probably say something about this?

Field access is very low-level, so I won't do this without making sure performance is preserved.

After all I'm not convinced overloading fields is a good idea. With this proposal, there would always be two ways of setting a property: x.property = value and property!(x, value). If field overloading is implemented, we'll need a very strong style guide to avoid ending in a total mess where you never know in advance which solution the author has chosen for a given type.

And then there would be the question of whether fields are public or private. Not allowing field overloading would make the type system clearer: fields would always be private. Methods would be public, and types would be able to declare they implement interfaces/protocol/traits, i.e. that they provide a given set of methods. This would go against @stevengj's https://github.com/JuliaLang/julia/issues/1974#issuecomment-12083268 about overloading fields with methods to avoid breaking an API: only offer methods as part of the API, and never fields.

The only place where I would regret field overloading is for DataFrames, since df[:a] is not as nice as df.a. But that doesn't sound like it should require alone such a major change. The other use case seems to be PyCall, which may indicate that field overloading should be allowed, but only for highly specific, non-Julian use cases. But how to prevent people from misusing a feature once it's available? Hide it in a special module?

@nalimilan, I would say that the preference should be to use x.property syntax as much as possible. The thing is that people _really_ like this syntax – it is very pleasant. Taking such a nice syntax and specifically saying that it should only ever be used for internal access to objects seems downright perverse – "hah, this nice syntax exists; don't use it!" It seems much more reasonable to make the syntax to access private things less convenient and pretty instead of forcing APIs to use the uglier syntax. Perhaps this is a good use case for the .. operator: the private _real_ field access operator.

I actually think that this change can make things clearer and more consistent rather than less so. Consider ranges – currently there's a sort of hideous mix of step(r) versus r.step styles out there right now. Especially since I introduced FloatRange this is dangerous because only code that uses step(r) is correct. The reason for the mix is that some properties of ranges are stored and some are computed – but those have changed over time and are in fact different for different types of ranges. It would be better style if every access was of the step(r) style except the definition of step(r) itself. But there are some steep psychological barriers against that. If we make r.step a method call that defaults to r..step, then people can just do what they're naturally inclined to do.

To play devil's advocate (with myself), should we write r.length or length(r)? Inconsistency between generic functions and methods are a problem that has afflicted Python, while Ruby committed fully to the r.length style.

+1 for .. as Core.getfield!

@StefanKarpinski Makes sense, but then you'll need to add syntax for private fields, and interfaces will have to specify both methods and public fields. And indeed you need a style guide to ensure some consistency; the case of length is a difficult one, but then there is also e.g. size, which is very similar but needs a dimension index. This decision opens a can of worms...

In that case, I also support .. to access actual fields, and . to access fields, be they methods or real values.

To play devil's advocate (with myself), should we write r.length or length(r)? Inconsistency between generic functions and methods are a problem that has afflicted Python, while Ruby committed fully to the r.length style.

The key factor that may be disambiguating for this issue is whether you want to be able to use something as a higher order function or not. I.e. the f in f(x) is something you can map over a collection, whereas the f in x.f is not (short of writing x -> x.f) – which is the same situation for all methods in single-dispatch languages.

Why stop at field access? What about having x.foo(args...) equivalent to getfield(x::MyType, ::Field{:foo}, args...) = ... ? Then we could have x.size(1) for size along first dimension. (not sure whether I'm fond of my suggestion, but maybe something to consider. Or probably not, as people will just write OO look-alike code?)

That would be possible with this functionality. Which is one of the things that gives me pause. I don't have a problem with o.o. style code like that – as I said, it's fairly pleasant and people really like it – but it does introduce enough choice in ways to write things that we _really_ need a strong policy about what you should do since you'll be very free with what you can do.

When I started to learn Julia, the no-dot syntax helped me a lot to mentally let go of OO-programming style. So for that reason alone, I think that my suggestion is bad.

Also, for simple overloading (i.e. just a.b sans (args...)), I agree with @nalimilan's comment above. In issue #4935 the consensus seems to be that fields should not be part of the API but only methods; consequently it seems that that issue will be closed. Having the .-overloading syntax will make it much less clear that normal-fields should not be part of the API and will probably encourage to make fields part of the API.

But yes, the . syntax is convenient...

How about: the single . should _only_ be syntactic sugar for getfield(x::MyType, ::Field{:name}) = ... and field access is _only_ through .. (i.e. what . is now).

This would allow to make the clear distinction:

• the . is for public API to access value-like things of type-instances
• the .. is for field access and should generally not be used in the public API

Of course, this would be a breaking change.

That's basically what I was proposing, except that . defaults to .. so it's not breaking.

Sorry, I should have re-read!

But I think the . not defaulting to .. might actually be nice (apart from that it is breaking), as it would force a decision on the developer about what is public API and what not. Also, if the user uses a .. than he can expect that his code may break, whereas . should not.

That's a good point. We can go that route by having a.b default to a..b with a deprecation warning.

From a style perspective, I think I'd much prefer to see

a = [1:10]
a.length()
a.size()

than

a.length
a.size

I think it helps preserve the idea that a function is being called instead of just a property being retrieved that is somehow stored in the type (back to the "purity" concern above). I wonder if there's a way to help ensure this kind of style so things don't get as messy as it is in some other languages.

I don't really like

a.length()

since then I can't tell if there was a function field in the original type. If . never accesses fields, that's obviously not an issue. Otherwise, it seems confusing to me.

A priori, I feel that we shouldn't do either a.length() or a.length. But the question is why? What makes r.step different from r.length? Is it different? If they're not different, should we use step(r) and length(r) or r.step and r.length?

With the semantics suggested by Stefan and the addition by me it would be clear that . always is a function call (just like + too), whereas .. is always a field access.

On the issue whether a.length, etc is a good idea: how about . access should only be used to access actual data in the type, more or less as one would use the entries of a dict. Whereas we stick with functions for the none-data properties like, size, length, step etc. Because some of them will need extra parameters and, I think, the a.size(1) type of syntax is bad.

Here is my take on this topic:

• The dot syntax should only be used for attributes of a type/class. Please keep in mind that this is not only about getters but also setters and something like a.property() = ... feels completely wrong.
• While I kind of like the current situation where function define the public API and fields are private, I share Stefans opinion that the dot syntax is too nice to be forbidden for public APIs. But please lets restrict this to simple attributes. a.length is a good example, a.size(1) not because it requires an additional argument.
• Please let . default to ... Julia is not known to be a boilerplate language. Lets keep it that way

Please let . default to ... Julia is not known to be a boilerplate language. Lets keep it that way

I do tend to agree with this. The syntax for setting even a synthetic property would just be a.property = b, not a.property() = b.

Sure, I just wanted to make clear why a.property() as a syntax is IMHO not nice

Or more clearly: The important thing about the dot syntax is not that one can associate functions with types/classes but its the ability to write getters/setters in a nice way. And getters/setters are important for data encapsulation (keep the interface stable but change the implementation)

This change would be great from an API designers perspective but I agree that it should come with some sort of style guide to limit any future inconsistency.

This would enable Ruby like dsl's...

amt = 1.dollar + 2.dollars + 3.dollars.20.cents 

But be prepared for java like madness:

object.propert1.property2.property3 ....

Just a few thoughts:

• I most want the . syntax for Dicts with Symbols as keys. Its just nicer to use d.key then d[:key]. But in the end it's not critical.
• I think that a->property reads better than a..property. But again it is not that big a deal and I don't know if it would work with julia syntax.

@BobPortmann I disagree. A dictionary is a container object, the API for container objects is obj[index] or obj[key]. Right now because we don't have properties in Julia, the container API is overloaded to provide this functionality in libraries like PyCall and in OpenCL. This change helps to strengthen the distinction of the container API as it will not be overloaded to provide additional functionality.

Using a->property for private fields would be a good way to keep C hackers away from Julia ;-)

I kind of like the .. syntax.

The a->property syntax is already spoken for – that's an anonymous function. The a..b operator has been up for grabs for a while, however. There are some cases where you want something that's dict-like but has lots of optional fields. Using getter/setter syntax for that would be nicer than dict indexing syntax.

"The a->property syntax is already spoken for – that's an anonymous function."

Yes, of course. It didn't look like it without spaces around the ->.

As a style guideline, how about recommending that property(x) be used for read-only properties and that x.property be used for read/write properties?

For writable properties, x.foo = bar is really much nicer than set_foo!(x, bar).

Having foo(x) for reading and x.foo for writing is quite confusing. Actually this is what properties make so appealing. Having the same syntax for read and write access, i.e. the most simple syntax one can get (for getters and setters)

Regarding style there is the big question whether we want to have both x.length and length(x) if this feature gets implemented or whether the later form should be deprecated and removed.

My opinion is that we should only have one way of doing it and only use x.length in the future. And regarding style I think its quite simple. Everything that is a simple property of a type should be implemented using the field syntax. Everything else with functions. I have used properties in C# a lot and rarely found a case where I was unsure whether something should be a property or not.

I'm against changing a randomly-chosen set of 1-argument functions to x.f syntax. I think @mauro3 made a good point that doing this obscures the nature of the language.

a.b is, at least visually, kind of a scoping construct. The b need not be a globally-visible identifier. This is a crucial difference. For example, matrix factorizations with an upper part have a .U property, but this is not really a generic thing --- we don't want a global function U. Of course this is a bit subjective, especially since you can easily define U(x) = x.U. But length is a different kind of thing. It is more useful for it to be first class (e.g. map(length, lst)).

Here are the guidelines I would suggest. The foo.bar notation is appropriate when:

1. foo actually has a field named bar. Example: (1:10).start.
2. foo is an instance of a group of related types, some of which actually have a field named .bar; even if foo doesn't actually have a bar field, the value of that field is implied by its type. Examples: (1:10).step, (0.1:0.1:0.3).step.
3. Although foo doesn't explicitly store bar, it stores equivalent information in a more compact or efficient form that is less convenient to use. Example: lufact(rand(5,5)).U.
4. You are emulating an API from another like Python or Java.

It may make sense for the bar property to be assignable in cases 1 and 3 but not 2. In case 2, since you cannot change the type of a value, you cannot mutate the bar property that is implied by that type. In such cases, you probably want to disallow mutation of the bar property of the other related types, either by making them immutable or by explicitly making foo.bar = baz an error.

@tknopp, I wasn't suggesting using x.foo for writing and foo(x) for reading. My suggestion was that _if_ a property is both readable and writable, then probably you want to _both_ read and write it with x.foo.

@StefanKarpinski: But isn't length a case of 3. where the sizes are whats usually stored and length is the product of the sizes?

I see Jeffs point though that this change would make these functions not first class anymore.

@stevengj: I see. Sorry for confusing that.

@tknopp – the length is derived from the sizes, but not equivalent to them. If you know the sizes you can compute the length but not vice versa. Of course, this is a bit of a blurry line. The main reason this is acceptable for lufact is that we haven't figured out a better API than that. Another approach would be to define upper and lower generic functions that give the upper-triangular and lower-triangular parts of general matrices. However, this approach doesn't generalize to QR factorizations, for example.

It's telling that there are only a few cases that _really_ seem to ask for this syntax: pycall, factorizations, and maybe dataframes.
I'm quite worried about ending up with a random jumble of f(x) vs. x.f; it would make the system much harder to learn.

Doesn't point 1 of @StefanKarpinski's list mean that any field of a type automatically belongs to public API?

At the moment I can tell what is the public API of a module: all exported functions and types (but not their fields). After this change, it would not be possible to tell which fields are supposed to belong to the public API and which not. We could start naming private fields a._foo or so, like in python, but that seems not so nice.

Personally I think the DataFrames case is a little superfluous. If we do this, I'll add the functionality to DataFrames, but I find the loss of consistency much more troubling than saving a few characters.

I would also not make the decision dependent on DataFrames, PyCall (and Gtk wants it also). Either we want it because we think that fields should be part of a public interface (because it "looks nice") or we don't want it.

... pycall ...

and JavaCall

Since the main use case for this seems to be interactions with non-Julia systems, what about using the proposed .. operator instead of overloading .?

I wonder if a simpler solution here is a more general hat-tip to OO:

#we already do
A[b] => getindex(A,b)
#we could have
A.b(args...) => b(A, args...)
# while
A..b => getfield(A,::Field{:b})
# with default
getfield(A, ::Field{:b}) = getfield(A, :b)

It seems like this would allow JavaCall/PyCall to do method definitions "in" classes, while also allowing a general style if people want to have some OO type code, though it's very transparent A.b() is just a rewrite. I think this would be very natural for people coming from OO.
Also having the new getfield with A..b to allow overloading there, though overloading here is strongly discouraged and only to be used for field-like/properties (I suspect it wouldn't be used very widely due to the slight scariness of overloading getfield(A, ::Field{:field}).

@mauro3:

Doesn't point 1 of @StefanKarpinski's list mean that any field of a type automatically belongs to public API?

That was a list of when it's ok to use foo.bar notation, not when it's necessary. You can disable the foo.bar notation for "private" fields, which would then only be accessible via foo..bar.

@karbarcca: I'm not super clear on what you're proposing here.

fwiw, I'm a fan of taking the consenting-adults-by-convention approach and making . fully overloadable. I think the double-dot proposal would lead to more confusion rather than less.

@ihnorton – as in you're against using a..b as the (unoverloadble) core syntax for field access or against using a..b for the overloadable syntax?

One of julia's best features is its simplicity. Overloading x.y feels like the first step on the road to C++.

@StefanKarpinski but then this would mean quite a shift in paradigm from default private fields to default public fields.

A realization I just had, probably this was clear to others all along. Full OO-style programming can be done with the basic .-overloading (albeit it's ugly). Defining

getfield(x::MyType, ::Field{:foo}) = args -> foofun(x, args...) # a method, i.e. returns a function
getfield(x::MyType, ::Field{:bar}) = x..bar+2                  # field access, i.e. returns a value

then x.foo(a,b) and x.bar work. So the discussion on whether x.size(1) should be implemented or only x.size is moot.

@StefanKarpinski against generally overloadable a..b and lukewarm about a..b -> Core.getfield(a,b).

I do start to see the need for another operator here, but a..b is not quite convincing. Needing two characters feels very... second class. Maybe a@b, a$b, or a|b (bitwise operators are just not used that often). An outside possibility is also ab`, which the parser could probably distinguish from commands.

I'd be ok with using the "ugly" operator for primitive field access. I think experience has shown that since it is a concrete operation it is rarely used, and indeed somewhat dangerous to use.

I'm suggesting allowing simulating OO single dispatch by the convention/rewriting:

type Type end
# I can define methods with my Type as 1st argument
method(T, args...) = # method body
t = Type()
# then I can call that method, exactly like Java/Python methods, via:
t.method(args...)
# so
t.method(args...) 
# is just a rewrite to
method(t, args...)

The justification here is we already do similar syntax rewrites for getindex/setindex!, so let's allow full OO syntax with this. That way, PyCall and JavaCall don't have to do

my_dna[:find]("ACT")
# they can do
my_dna.find("ACT")
# by defining the appropriate find( ::PyObject, args...) method when importing modules from Python/Java

I like this because it's a fairly clear transformation, just like getindex/setindex, but allows simulating a single dispatch OO system if desired, particularly for OO language packages.

I was then suggesting the use of the .. operator for field access, with the option to overload. The use here would be allowing PyCall/JavaCall to simulate field access by overloading calls to .., allowing DataFrames to overload .. for column access, etc. This would also be the new default field access in general for any type.

I do have a soft spot for pure syntax rewrites. It's arguably a bad thing that you can write a.f(x) right now and have it work but mean something confusingly different than most OO languages.

Of course the other side of that coin is horrible style fragmentation, and the fact that a.f has nothing in common with a.f(), causing the illusion to break down quickly.

One of julia's best features is its simplicity. Overloading x.y feels like the first step on the road to C++.

Same feeling here. I was considering, if the actual need for this is really for a limited number of interop types, what about only making it valid if explicitly asked in the type declaration? E.g. an additional keyword besides type and immutable could be ootype or something.

and the fact that a.f has nothing in common with a.f(), causing the illusion to break down quickly.

Can you clarify what this means @JeffBezanson?

I'd expect that a.f is some kind of method object if a.f() works.

Ah, got it. Yeah, you definitely wouldn't be able to do something like map(t.method,collection).

I'm going to agree with @mauro3 that by allowing obj.method(...), there is a risk that new users may just see julia as another object-oriented language trying to compete with python, ruby etc., and not fully appreciate the awesomeness that is multiple-dispatch. The other risk is that standard oo style then become predominant, as this is what users are more familiar with, as opposed to the more julian style developed so far.

Since the use case, other than DataFrames, is restricted to inter-op with oo languages, could this just all be handled by macros? i.e. @oo obj.method(a) becomes method(obj,a)?

@karbarcca this would mean that automatically everything could be written in two ways:

x = 3
x.sin()
sin(x)
x + 2
x.+(2) # ?!

@karbarcca https://github.com/JuliaLang/julia/issues/1974#issuecomment-38830330

t.method(args...)
# is just a rewrite to
method(t, args...)

That would not be necessary to PyCall since the overloadable dot could just be used to call pyobj[:func] by pyobj.func. Then pyobj.func() would be in fact (pyobj.func)() .

Rewriting a.foo(x) as foo(a, x) would not solve the problem for PyCall, because foo isn't and cannot be a Julia method, it is something I need to look up dynamically at runtime. I need to rewrite a.foo(x) as getfield(a, Field{:foo})(x) or similar [or possibly as getfield(a, Field{:foo}, x)] so that my getfield{S}(::PyObject, ::Type{Field{S}}) can do the right thing.

@JeffBezanson https://github.com/JuliaLang/julia/issues/1974#issuecomment-38837755

I do start to see the need for another operator here, but a..b is not quite convincing. Needing two characters feels very... second class

I would say that, on the other hand, .. is typed much more quickly than $, @ or | as no shift key needs to be pressed, and while being two characters the finger stays on the same key :smile:

@stevengj Ah, I see. But my point still stands, that the rewriting could be done with a macro.

For JavaCall, I actually only need essentially a unknownProperty handler. I dont actually need to rewrite or intercept existing property read or write. So would a rule that "a.x gets re-written to getfield(a, :x) only when x is not an existing property" help keep things sane?

@simonbyrne, requiring a macro would defeat the desire for clean and transparent interlanguage calling. Also, it would be hard to make it work reliably. For example, suppose that you have a type Foo; p::PyObject; end, and for an object f::Foo you want to do foo.p.bar where bar is a Python property lookup. It's hard to imagine a macro that could reliably distinguish the meanings of the two dots in foo.p.bar.

Honestly, I don't see the big deal with style. High-quality packages will imitate the style of Base and other packages where possible, and some people will write weird code no matter what we do. If we put dot overloading in a later section of the manual, and recommend its use only in a few carefully selected cases (e.g. inter-language interoperability, read/write properties, maybe for avoiding namespace pollution for things like factor.U, and in general as a cleaner alternative to foo[:bar]), then I don't think we'll be overrun with packages using dot for everything. The main thing is to decide what _we_ will use and recommend this for, and probably we should keep the list of recommended uses very short and only extend it as real-world needs arise.

We're not adding super-easy OO-like syntax like type Foo; bar(...) = ....; end for foo.bar(...), so that will limit temptation for newbies too.

I'm basically in full agreement with @stevengj here. I like a..b for real field access because it

1. looks similar to a.b
2. is less convenient, as it should be
3. is only _slightly_ less convenient
4. has no existing meaning and we haven't found any compelling use for it in over a year
5. isn't horrifically weird like ab`

With this change and possibly (https://github.com/JuliaLang/julia/issues/2403) will nearly all of Julia's syntax be overloadable? (The ternary operator is the only exception I can think of) That almost all syntax is lowered to overloadable method dispatch seems to be a strongly unifying feature to me.

I agree that it's actually kind of a simplification. The ternary operator and && and || are really control flow, so that's kind of different. Of course that kind of argues against making a..b the real field access since then _that_ would be the only non-overloadable syntax. But I still think it's a good idea. Consistency is good but not paramount for its own sake.

Oh, there's also function call which is not overloadable. So basic I forgot about it.

That is what issue #2403 addresses.

Yep. But this is a lot closer to happening than that is.

The only fly in the ointment for me here is that it would be really nice to use the real field access operator for modules, but that probably won't happen since nobody wants to write Package..foo.

Tab-completing after dots gets a bit ugly; technically you have to check what method x. might call to see if it's appropriate to list object field names or module names. And I hope nobody tries to define getfield(::Module, ...).

I think that tab completing can be done like this: foo.<tab> lists the "public fields" and foo..<tab> lists the "private fields". For modules, would it be ok to just allow the default implementation of Mod.foo be Mod..foo and just tell people not to add getfield methods to Module? I mean, you can already redefine integer addition in the language – all hell breaks loose and you get a segfault but we don't try to prevent it. This can't be worse than that, can it?

It is in fact slightly worse than that, because a programming language really only cares about naming. Resolving names is much more important than adding integers.

We don't have much choice but to have Mod.foo default to Mod..foo, but we'll probably have to use Mod..foo for bootstrapping in some places. The .. operator is extremely helpful here, since without it you can't even call Core.getfield in order to define the fallback. With it, we'd probably just remove Core.getfield and only have ...

That's a fair point – naming is kind of a big deal in programming :-). Seems like a good way to go – only .. and no Core.getfield.

This two ideas,

[...] put dot overloading in a later section of the manual, and recommend its use only in a few carefully selected cases @stevengj https://github.com/JuliaLang/julia/issues/1974#issuecomment-38847340

and

[...] the preference should be to use x.property syntax as much as possible @StefanKarpinski https://github.com/JuliaLang/julia/issues/1974#issuecomment-38694885

are clearly opposed.

I think that if the first idea is to be chosen then just creating a new .. operator for those "carefully selected cases" makes more sense.
As advantage, using ..name for cases where currently [:name] is used (DataFrames, Dict{Symbol, ...}) would be more typing/syntax friendly while clearly stating that something different from field access was happening. Moreover, the double dot in ..name could be seen as a rotated colon, a hint to the symbol syntax :name, and also there would be no problem with tab completions.
As disadvantage, the uses in PyCall et al. would be not so close to the original syntaxes (and could even be confusing for the cases when the . really must be used). But let's be honest, Julia will never be fully Python syntax compatible, and there will always be cases where one has to type a lot in Julia with PyCall to perform otherwise simple instructions in Python. The .. to emulate . could give a good balance here. (Please don't get me wrong, I really like PyCall and think it is a critical feature which deserves special care)

The second ideia, which I currently prefer, has the big decision about when property(x) or x.property must be used, which requires an elegant, well though, and clear definition, if such thing exists...
It seems that if people want an overloadable . that's because they prefer x.property API style in the first place though.
Anyway, I would prefer to see . not as a overloadable field access operator but as a overloadable "property" access operator (getprop(a, Field{:foo}) maybe?) which defaults to a non-overloadable field operator ...
Other decisions would also have to be taken, e.g., which will be used in concrete implementation code for field access, .. or .? For example, for the Ranges step example, which will be idiomatic? step(r::Range1) = one(r..start) or step(r::Range1) = one(r.start)? (not to mention the question whether step must be a method or a property).

That's why I backed off of that angle and proposed these criteria: https://github.com/JuliaLang/julia/issues/1974#issuecomment-38812139.

Just one thought that popped in to my head while reading this interesting thread. Export could be used to declare public fields, while all fields are visible inside the defining module, eg:

module Foo
   type Person
     name
     age
   end
   export Person, Person.name
   @property Person :age(person) = person..age + 1
end

In this situation the exported Person still looks like 'name' and 'age' except in this case age is readonly through a function that adds one. Exporting all of Person might be done as export Person.* or similar.

[pao: quotes]

@emeseles Please be careful to use backticks to quote things that are like Julia code--this ensures formatting is maintained, and prevents Julia's macros from creating GitHub notifications for similarly-named users.

. and .. are confusing: a clear and easy to remember sintax is something good

I'm really looking forward to being able to do this. Is this a big enough change to get it (or the WIP in #5848) flagged as a 0.4-project?

Yep, it's definitely a project.

I think most of us agree that its recommended usages should be limited, at least to start with. My feeling is that it should be initially recommended for only two usages: interoperability (with other languages, as in PyCall, and more generally for external libraries where dot notation is natural), and perhaps for objects with mutable state (since get_foo(x) and set_foo!(x, val) are ugly).

Even if we recommend it only for foreign-call interoperability, that purpose alone is enough to justify this feature in my opinion. For a new language like Julia, talking smoothly with the rest of the software universe is hugely important.

Steven, I am not a 100% sure about the getter/setter because I fear that it will soon lead to inconsistencies but I agree with the other use case. Ontop of that we have in Gtk.jl dynamic properties which would also benefit from the syntax. My personal favorite is the enum implementation that Stefan outlined in #5842 though.

Bump. What is blocking the progress on this issue? Is a decision needed, or is this issue depend on other internal changes, not yet done, or is it just coding?

What is blocking the progress on this issue?

Someone doing the work and some hesitation about whether it's the right thing to do.

Note that @ihnorton already made an early draft implementation at #5848. I think work has stalled primarily because a clear statement from the Julia core team is needed on whether this is a desired feature.

I'm on board with this. @JeffBezanson seems to be on the fence.

For me, having this feature, would make a transition from our large Python code base to Julia easier. To explain students, if they use Python code they need a quite different syntax then they are used to, this might become difficult.

We had this discussion above in this thread and I still cannot see full agreement. Currently several people think that a public API is made of functions/methods while the private API is the fields of a composite type. I can see very rare exceptions from this scheme. (.U in an LU decomposition?)

This does not mean I am against this because Python access and enums are cases where this is useful. Still one can question how urgent the need here is and if it would be wise to push this in the end of a dev cycle.

@ufechner7, I agree that the main motivation is inter-language interop. @tknopp, we are never going to get unanimous agreement on something like this. Ultimately it comes down to what @JeffBezanson and @StefanKarpinski decide.

I think a lot of the hesitation stems from what I imagine may be Jeff's worst nightmare:

module DotOrientedProgramming
  Base.getfield(x, ::Field{:size}) = size(x)
  Base.getfield(x, ::Field{:length}) = length(x)
  ⋮
end

I would very much dislike this, too - any package that decides to misuse it like this will impose their misuse on _all_ types in the system, including my own. This feature is very powerful and will change how Julia is written. For better and (perhaps, but hopefully not) worse.

Yes sure Steven, that may not be properly worded from me. The point I wanted to make is that this change can have a major influence in how the language will evolve. And the "formal interface" idea that we have in another issue is also influence by making . overloadable. So yes lets @JeffBezanson and @StefanKarpinski decide. Still the question is if the decision has to be enforced now...

For what it's worth, I've come to favor making almost all syntax overloadable and then relying on culture to resist going too hog wild with it.

+1. I think there is a strong philosophical (and possibly practical...) analog to call overloading here. The manual needs a section entitled Don't do stupid stuff: we won't optimize that. (of course, call overloading was partly _for_ performance reasons -- but it is rife with potential for abuse)

• :100: to that. I think culture should be enough motivation to not abuse this

Overall, I am in favor. Potential for abuse is not my biggest worry. For me the big problems are

• Acceptable syntax for "real" field access. I don't like a..b all that much.
• Modules. Qualified names are extremely important. Expressing them with method dispatch is possible, but has practical difficulties. Namely, you need to go through many compiler phases (perhaps even through inlining) just to know you have a qualified name. This makes life harder for anybody writing tools that consume ASTs. It also makes it very easy to handle this case incorrectly in such tools.

These problems could be solved in one stroke by using the same syntax for both, but it's nearly impossible to imagine using anything but . for modules at this point. _Internally_ there will definitely be abstract syntax for module references; it would be frustrating if there were no nice way to expose that.

My two cents on this question : why not use : for qualified names ? It is already in use for something similar:

import Base: call, show, size

This would give something like

module Foo
    module Bar
        f(x) = 3*x
    end
    const a = 42
end

@assert Foo:a == 42

Foo:Bar:f(789)

Or is their already too much uses of the : symbol ? The :: symbol (C++ style) seems to be too much verbose for me.

The : is already the most overloaded symbol in Julia, so that's not going to help, I'm afraid.

Can we simplify the qualified naming issue by making module.name not overloadable? Since module bindings are constant, that would allow us to keep the same semantics but short-circuit all of the normal logic for qualified name lookups just as soon as it's known that the LHS of a.b is a module. I think it's pretty reasonable to not allow people to override what it means to look a name up in a module.

I rather like the a..b syntax for real field access. What's your objection to it?

Aside: I kind of wish we had gone with ( ) for import lists like some of the functional languages. I.e.:

import Base (call, show, size)

My reason is that we could make the commas optional and allow trailing commas. It really annoys me that all of the imported names need trailing commas except the last one which cannot have one.

Yes, I was just about to mention the possibility of making a.b mean "if a is a module then do module lookup first". That might help, and we certainly don't want to override the meaning of module lookup. It does have some complexity cost though, since we then can't represent a.b as the call getfield(a,:b). It needs to be a special AST node with an implicit branch. Of course we could use an _explicit_ branch, but I'd worry about AST bloat from that.

There doesn't seem to be an easy way out of such a huge conflict between the needs of the front end and back end.

If everybody else likes a..b I guess I can learn to live with it. It just looks to me like it means something totally different, an interval perhaps.

I dislike a..b as well but wonder why it would be required at all. When reading this thread one gets the impression that overloading will only be used in language wrappers and dynamic use cases where the real field access is not required.

Because at some point you need to access the representation of an object in order to do anything with it. One could argue that this would be relatively rare, and so can be ugly like get_actual_field(a,:x), but this seems like too important an operation not to have syntax.

I see that but this sounds like we seek for a syntax we want nobody to use right?

Not providing .. would be a way to say yes for dynamic use cases but no for dot-oriented programming

I don't see how that would prevent dot-oriented programming; you could still do @mbauman 's example above.

While the a..b syntax does kind of look like an interval (I've used it as such), I just don't think that interval arithmetic needs its own input syntax – writing Interval(a,b) is just fine and there isn't much else anyone wants to use that syntax for, since it's been an operator in Julia for years now and no one is using it for much of anything. It also kind of looks like field access.

One silver lining to this is we can replace the hideous module_name with m..name. Not being able to access the fields of Module objects has been a wart.

Yes, I was just about to mention the possibility of making a.b mean "if a is a module then do module lookup first". That might help, and we certainly don't want to override the meaning of module lookup. It does have some complexity cost though, since we then can't represent a.b as the call getfield(a,:b). It needs to be a special AST node with an implicit branch. Of course we could use an _explicit_ branch, but I'd worry about AST bloat from that.

Could we handle this by making a.b unconditionally mean getfield(a,:b) and then making it an error to add methods to getfield that intersect the getfield(::Module, ::Field) method? It's sort of an odd way to enforce that behavior, but it would ultimately have the same effect. Then lowering could just use that fact that we know you can't do that to cheat and lower module.name to qualified name lookup.

Ok I state it the other way around: Would anybody in this thread use .. and if yes what would be an exemplary use case? (i.e. might entirely shadowing the internal field access be ok)

@StefanKarpinski Yes, that might work. Could be another case where we want some kind of "sealed" methods.

@tknopp Accessing module..name and module..parent :) Also, just to clarify, are you advocating function-call syntax like get(obj,:field) for low-level field access?

No I am not advocating a certain syntax. I just think it would be good to make sure why this feature is needed and what the use cases are. For the dynamic use cases it would be ok that

• a.b is a field access if Base.getfield(a, ::Field{:b}) has not been defined
• a.b is the dynamic version if Base.getfield(a, ::Field{:b}) is defined. In this case the real field access could be shadowed

My question was if there are use cases where shadowing is not ok.

Yes; you might want to define pyobject.x so that x is always looked up in the pyobject's dictionary, for all x. Then a separate mechanism is needed to access the pyobject's julia fields.

Ahhh, so its all or nothing? I somehow got the impression that one could have

type A
  c
end

Base.getfield(a::A, ::Field{:b}) = 3

a = A(1)

a.c # This still calls the field access
a.b # This calls the function

Yes, you _can_ do that, but not all objects will. Some will want to define getfield(a::A, ::Field) to intercept all fields.

Ok thanks now I get it. All the dynamic use cases would want getfield(a::A, ::Field) and thus need any way to call the internal fields.

Then my take is that Core.getfield is sufficient unless someone finds a practical use case where this is annoying.

This is probably a given, but we're also going to allow overriding setfield!, right? I'd really like that for exposing mutable views into a database in which rows become types.

Yes, that was my impression.

Ok, IMHO whether to use .. for real field access or Core.getfield is not such a big deal. One could introduce the general feature as experimental and make this subject to change.

The question is whether this will fit into the time frame of 0.4 or not. So it #5848 close to the final implementation and the module thingy solvable?

@johnmyleswhite: I would also vote for making this symmetric and also allowing setfield!. In Gtk.jl we would use both.

It doesn't seem very clear what would be the rule when to use this feature and when not to use it. I see the point for PyCall, where a method/field must be dynamically looked up, and thus cannot be a Julia method/composite type (and the resulting syntax is closer to Python). But then, why use it for Gtk.jl? If it starts doing foo.bar = x instead of setbar!(foo, x), then standard Julia code will invitably start using this pattern too: is this what we want? Maybe it is, but let's be clear about that.

Would it be acceptable/recommended to use this feature to implement property getters and setters defined for abstract (and concrete too) types?
I guess that would allow the avoidance of name clash of methods which are used to get properties from different types of different modules.

Ref.: https://github.com/JuliaLang/julia/issues/4345, https://groups.google.com/forum/#!msg/julia-users/p5-lVNlDC8U/6PYcvvsg29UJ

@nalimilan: Gtk has a dynamic property system its not about getters/setters.

@tknopp Ah, OK, indeed. But for most common properties you have a (fast) getter/setter function, plus the dynamic property. So would you recommend using the getter/setter function when available, and the field overloading syntax only for properties that don't have one? Sounds fine to me -- but it's good to have a clear policy about this IMHO.

In my view at this point (and I think we need to experiment with this a bit to figure out the right rules), f(x) is better when f makes sense as a general standalone concept like "length" while x.f should be used when f is not really independent of x. To try to fit my previous example into that, it's not really useful to have a generic step function since most vectors and collections don't have any kind of notion of step – it _only_ makes sense when you have a range of some kind. Thus, it's ok to have x.step be the way to get the step of a range x. It's a bit of a judgement call, but I guess life is full of those.

I don't like .. as it doesn't convey direct access to me. How about foo.bar. The extra dot at the end pins it to be direct access.

Could also pick a unicode symbol: we still have lots of those left...

@GlenHertz, that doesn't really work if you have to chain field accesses.

@simonbyrne, I'm generally against having anything in the core language or standard library that requires the use of Unicode. Allowing it is one thing, forcing people to use it is another entirely.

In my view at this point (and I think we need to experiment with this a bit to figure out the right rules), f(x) is better when f makes sense as a general standalone concept like "length" while x.f should be used when f is not really independent of x.

My personal rule for using this feature is going to be: only use this for language interop or for things that are either "almost fields" or "augmented fields". For example, I might use this to update a cache that depends on the value of all of the fields in a type.

One big question I have about this feature: how does this interact with type inference? It seems like you're about to define a function getfield(x::T, s::Symbol) that produces differently typed output for different values of s. Does that only work because getfield is magical? Can you redefine the output of getfield(x, s) for fixed x and s at any point in a program? If so, how does that mesh with the inability to redefine a type?

It seems like you're about to define a function getfield(x::T, s::Symbol) that produces differently typed output for different values of s.

That's why the plan is to express this as getfield{s}(x::T, f::Field{s}) where s is a symbol.

I had missed that. Thanks for setting me straight.

@nalimilan: Yes, the overloaded fields would only be used for the dynamic properties. This is how Jameson wants to tackle this and I think this is good. All the real getters and setters are autogenerated but still functions without all the get/set naming. The live in the GAccessor module (short G_)

On the syntax, why not use <- for real field access?
It's similar to -> in c++ which is in use for lamdas but <- is currently unused.
It could be read as from the type, get the value directly.

It would leave .. unused for those wanting to use it on intervals still and would be using up an unused pair which has no other uses I can think of so far.

Let's not use R's assignment notation for field access.

We could possibly use -> to mirror C/C++ directly and get new syntax for
anonymous functions. I’ve never much cared for the anonymous function
syntax since it’s a little terse/unreadable. Maybe we could instead do
something like

func (x) x^2 end

or the longer, more consistent

function (x) x^2 end

Perhaps there’d be a way to come up with a good syntax that doesn’t require
using an end.

Not to change the discussion too much, but it would definitely make sense
to use -> for real field access.
​

On Wed, Jan 28, 2015 at 8:49 AM, John Myles White [email protected]
wrote:

Let's not use R's assignment notation for field access.

—
Reply to this email directly or view it on GitHub
https://github.com/JuliaLang/julia/issues/1974#issuecomment-71857083.

@quinnj: func (x) x^2 end already works. But it is nice to have very concise syntax for anonymous functions: map(x->x^2, 1:10)

I don't think field access needs special syntax (a unicode character as @simonbyrne suggested is okay as on option). I wouldn't want to lose x -> x^2.

It looks like this issue is still open/pending discussion? Been interesting reading all of the various comments here about the dot operator.

Has there been any suggestions adding other new operator tokens? Using something like :> might be a nice alternative. It has parallels to |> and might have a more native Julia _feel_ to it:

c = foo.a + foo.b
pyobj:>obj:>process(c)

While still being far easier to write than:

pyobj[:obj][:procees](c)

Or comparing:

someobj :> array |> length 
# vs
length(get_array(someobj)) 

I'm new to Julia, but I've quickly gained a strong appreciation for the multiple dispatch approach. Object oriented programming – particularly for scientific programming – makes a lot of tasks more cumbersome. I'd be worried that the OO paradigm/syntax would negatively affect the development of Julia's style.

Or alternatively, since I forgot about string interning and/or quoting:

someobj <: field_name |> length

@elcritch, <: is currently used as Julia's "is subtype of" operator, and if :> is introduced it is likely that we'll want to use it for something type-related due to that heritage.

If using instance..member is a problem, here are some possibilities. Shield your eyes! It is likely that every one of these is worse:

• instance^.member (carrot dot)
• instance~.member (tilde dot)
• instance:.member (colon dot)
• instance*.member (star dot)
• instance-.member (dash dot)
• [email protected] (at-sign dot)
• instance&.member (amper dot)
• instance$.member (dollar dot)
• instance<.>member (spaceship dot)

I honestly think that (a) .. seems good enough and (b) it doesn't really matter whether it looks nice because this will always be an obscure corner of the language. Most people will use instance.member because they will only have either a field or a getfield method, but not both.

(For that matter, most people who want to use both a member and a method will probably not even bother to learn about ... They will just define a method for foo.member and name the "real" field foo._member. Arguably, this is better style anyway — it means that when you read the type definition, it will be immediately obvious that _member is not supposed to be something you can or should access directly. This would argue for making .. something ugly and obscure like :. rather than taking up valuable punctuation real-estate.)

I would miss the ability to use .. as an infix interval operator, but overloadable field access is a worthwhile trade-off. While I'm hesitant to mildly terrified of adding any more meanings to colon, :. doesn't seem that bad.

Note that :. is actually valid syntax for symbol(".") right now, so it might not be good to use that. The point that .. is potentially useful is well taken; we shouldn't waste it on a syntax hardly anyone will use. I'd be perfectly happy to go with something even uglier like @. (since . is not a valid macro name nor can it begin an identifier, this doesn't seem like to conflict with anything). Again, this is going to be such an obscure corner of Julia that it is not worthwhile trying to make it pretty.

+1 to just getting this done using .. and ignoring any potential ugliness

Yeah, let's go for .. anyway if there could be use for .. then I think it would be a _range_ constructor, but hey its already there with colon eg. start:stop.

One last punt: what about.: ? Is it too subtle to have a.b, a.(b), a.(:b) _and_ a.:b ?

@hayd, that seems too subtle and easy to use by accident.

@ihnorton, is there any chance of resurrecting a version of #5848? We could punt on the syntax question and just use Core.getfield(x, Field{y}) to access the "real" field.

Bikeshed about Core.getfield syntax aside, are there any substantive questions remaining?

In #5848, @tknopp suggested only making "real" field access overloadable, contrary to @JeffBezanson's suggestion that everything should be overloadable. Personally, I would be happy with making it impossible to overload real fields, except that the dynamic nature of the language will probably make that much more complicated to implement. e.g. with the "everything-overloadable" approach, if you have x::Vector{Any}, then doing x[i].y can be interpreted getfield(x[i], Field{:y}) and the dispatch system will do the right thing regardless of whether y is a real field, whereas if you only want to call getfield for "virtual" fields then the codegen will have to implement a miniature subset of the dispatch system for runtime checking of the x[i] type.

Another question was whether Module.foo should be overloadable. On the one hand, there is a certain consistency to using getfield for everything, and the abovementioned Vector{Any} example could have Module array members so we'd have to handle that case anyway. On the other hand @JeffBezanson pointed out that this could make compilation harder, and make the behavior of declarations like function Base.sum(...) hard to grok. My preference would be to make Module.foo non-overloadable, at least for now, in any case where the compiler knows it is working with a Module (i.e. not a Vector{Any}); the slight inconsistency seems worth it in order to be conservative about what gets changed.

+1 to not allow overloading of Module.foo.

To chime in here, one area of scientific computing where OO programming and syntax is actually superior to FP is agent based modeling. Although I miss concrete and multiple inheritance to set up agent hierarchies, the lightweight and fast abstractions and quick prototyping of Julia is amazing- Already a couple of ABM frameworks have popped up.

In ABM, the dot notation is preferable for expressing agent interactions: Agent1.dosomething(Agent2) vs dosomething(Agent1,Agent2).

This obvious isn't the biggest use case, but it would be nice to keep this syntactic sugar for thinking and coding about ABMs.

I would also very much like to have this syntax available in Julia. As
much as I appreciate the function oriented approach from a design
perspective, method call syntax is very useful and readable in several
domains. It would be great if A.b(C) was equivalent to b(A, C).
On Apr 22, 2015 8:50 AM, "datnamer" [email protected] wrote:

To chime in here, one area of scientific computing where OO programming
and syntax is actually superior to FP is agent based modeling. Although I
miss concrete and multiple inheritance to set up agent hierarchies, the
lightweight and fast abstractions and quick prototyping of Julia is
amazing- Already a couple of ABM frameworks have popped up.

In ABM, the dot notation is preferable for expressing agent interactions:
Agent1.dosomething(Agent2) vs dosomething(Agent1,Agent2).

This obvious isn't the biggest use case, but it would be nice to keep this
syntactic sugar for thinking and coding about ABMs.

—
Reply to this email directly or view it on GitHub
https://github.com/JuliaLang/julia/issues/1974#issuecomment-95218555.

In ABM, the dot notation is preferable for expressing agent interactions: Agent1.dosomething(Agent2) vs dosomething(Agent1,Agent2).

Why is that better? Edit: I mean in this ABM-context specifically.

Please, let's not get sucked into religious wars over spelling. @dbeach24, no one is proposing that a.b(c) be equivalent in Julia to b(a,c); that's not going to happen.

Overloadable dots is crucial for natural interop with other languages. That's reason enough.

Subject.Verb(DirectObject)

Is fairly natural in several contexts. A lot of OO programmers are use to
it, and while it is a mere reordering of function(A, B), that reordering
does a lot for readability, IMO.
On Apr 22, 2015 10:32 AM, "Andy Hayden" [email protected] wrote:

In ABM, the dot notation is preferable for expressing agent interactions:
Agent1.dosomething(Agent2) vs dosomething(Agent1,Agent2).

Why is that better?

—
Reply to this email directly or view it on GitHub
https://github.com/JuliaLang/julia/issues/1974#issuecomment-95256390.

I was proposing it. (Sorry, I didn't mean to start a war, nor did I know
the suggestion would be unpopular.) I thought I'd seen this come up before
in the forums, but did not realize that it had already been dismissed as a
bad idea. May I ask why? (Can you point me to a thread?)

Thanks.
On Apr 22, 2015 11:09 AM, "Steven G. Johnson" [email protected]
wrote:

Please, let's not get sucked into religious wars over spelling. @dbeach24
https://github.com/dbeach24, no one is proposing that a.b(c) be
equivalent in Julia to b(a,c); that's not going to happen.

Overloadable dots is crucial for smooth interop with other languages.
That's reason enough.

—
Reply to this email directly or view it on GitHub
https://github.com/JuliaLang/julia/issues/1974#issuecomment-95266671.

One reason not to do this is that a.b looks up b in the scope of a, while b alone looks up b in the enclosing scope. It would be very confusing if dotted access would sometimes not look up in the left side object.

Btw, it is considered a feature that functions in Julia are not looked up inside objects but in the current scope. I believe the fear that people would start to use functions looked up inside of objects is one reason that had been holding dot overloading back.

@toivoh, any implementation of dot overloading would use the existing method dispatch, so it wouldn't change the scoping behavior. @dbeach24, the basic reason not to encourage indiscriminate a.b(c) usage is that if you have too many syntaxes to do the same thing, the language and libraries turn into a mess. It's better to pick one spelling and stick with it, and Julia's multiple dispatch favors b(a,c) because it is clearer that b is not "owned" by a — the b(a,c) method is determined equally by _both_ a and c.

The biggest exception, of course, is for calling external libraries in languages like Python or C++, where it is nice to be able to mirror the dot syntax of the library you are calling. (So that e.g. people can translate documentation and examples directly into Julia without changing much.)

Am I all wet, but wouldn't a.b(arg) mean take an anonymous function stored in field b of a, and then evaluate it with the given argument?

Sent from my iPhone

On Apr 22, 2015, at 5:06 PM, Steven G. Johnson [email protected] wrote:

external

@ScottPJones That currently works just fine, but is generally not considered good style.

I wasn’t concerned about style or not, I just thought that since it already had a meaning, that was consistent with the way Julia works (i.e., being able to store anonymous functions in fields), that it was a good argument _not_ to try to treat a.b(arg) as if it were b(a,arg).
I _might_ have a use for having a struct (type) with members storing anonymous functions though, where I am loading functions written in Julia from a database, and then doing parse on them, and storing the functions into an object...
What would be a better “Julian” style to do something like that?

Thanks!

@ScottPJones I think there is agreement these shouldn't be equivalent*.

There can be exceptions to the "style" rule, but there has to be a compelling case to use put a function in field, same as for dot overloading. I think the issue is that people shouldn't do these willy-nilly / for the sake of it/because they can.

That might be an example, but there might also be a better way (certainly it's not the only way)...

99%+ of the time it's better to dispatch on typeof(a); no function-fields, no dot overloading.

*_However, I think everyone knows the second this lands there'll be a package that does just that..._

In D they even have a name "uniform function call syntax" for a.b(arg) and it's quite popular, but I think it's pretty deeply at-odds with the generic function, multiple dispatch way Julia works. If the functions in question are anonymous or completely duck-typed, then I suppose things will work, but that's pretty restrictive IMO. I don't think there's much reason to store function fields inside a composite type, except out of habit from class-based traditional OO languages. A better place to store generic functions if you're loading them from somewhere would be in modules.

But we've gotten pretty far from "substantive questions" now. I'm also in favor of being conservative with how we allow overloading getfield, not allowing getfield overloading on modules, and not worrying too much about special syntax for "true getfield."

@stevengj: Yes, and as I was trying to say, that is one fundamental reason why it's never going to happen that a.b(c) becomes equal to b(a, c), regardless of what we do with dot overloading otherwise.

There was some discussion on the mailing list touching on this. From my perspective the most relevant post (by @nalimilan) to this thread is: https://groups.google.com/d/msg/julia-users/yC-sw9ykZwM/-607E_FPtl0J

Adding to @johnmyleswhite's comment regarding personal policy on when to use this feature - it strikes me that some HTTP ideas could be useful here, and that getfield() should not have side-effects and setfield!() should be idempotent(that is, calling it multiple times with the same value should have the same effect as calling it once). Not necessarily hard-and-fast rules that are enforced by the compiler, but usage guidelines to keep things from getting too crazy.

I have posted a workaround using parametric types with pointer parameters and convert to call a custom setter when setting a field:
post: https://groups.google.com/forum/#!topic/julia-users/_I0VosEGa8o
code: https://github.com/barche/CppWrapper/blob/master/test/property.jl

I'm wondering if I should use this approach in my package as a workaround until setfield! overloading is available, or is it too much stress on the parametric type system?

I'd like to mention one additional benefit of getfield/setfield! overloading, I hope this is the right place for this, I'm sorry otherwise. (A related topic came up on https://groups.google.com/forum/#!topic/julia-users/ThQyCUgWb_Q )

Julia with getfield/setfield! overloading would allow for a surprisingly elegant implementation of autoreload functionality in an _external package_. (See all the hard work that had to be put into the IPython autoreload extension https://ipython.org/ipython-doc/3/config/extensions/autoreload.html to get this functionality.) The idea of autoreload is that you can modify functions and types in external modules while working with the REPL.

TLDR: getfield/setfield! overloading, dictionaries and a package similar to https://github.com/malmaud/Autoreload.jl should do the trick.

To be more specific, imagine a package similar to Autoreload.jl that does the following.

You first create a module M.jl:

module M
type Foo
  field1::Int64
end
bar(x::Foo) = x.field1 + 1.0
end

In the REPL, you type

julia> using Autoreload2
julia> arequire("M")
julia> foo = Foo(42)

Then you change M.jl to

module M
type Foo
  field1::Int64
  field2::Float64
end
bar(x::Foo) = x.field1+x.field2

This would get autoreloaded and transformed to

# type redefinition removed as already done by Autoreload.jl
const field2_dict = Dict{UInt64,Float64}()
setfield!(x::Foo, ::Field{:field2}, value) = field2_dict[object_id(x)] = value
getfield(x::Foo, ::Field{:field2}) = field2_dict[object_id(x)]
@do_not_inline bar(x::Foo) = x.field1 + x.field2

and then in the REPL you could do

julia> foo.field2 = 3.14
julia> println(bar(foo)) # prints 45.14

Performance wouldn't be worse than with Python, so people who migrate their workflow from IPython autoreload wouldn't lose anything in terms of performance. And once you restart the REPL, you're back to full performance.

I got tired of writing a[:field][:field2][:morestuff](b[:random_stuff]) as it is not really readable. So I wrote this little macro which works for my use-cases in 0.4 and 0.5
https://github.com/sneusse/DotOverload.jl

TL;DR
A macro which transforms the AST of an expression a.b -> getMember(a, :b)

Removing from 0.6, since there's no consensus that this is a good idea and there's a conflicting proposal for what to do with dot syntax.

@Keno: Do you have a link to the conflicting proposal?

Don't think @StefanKarpinski has written it up yet, but I'd expect there to be a Julep about it soon.

I found object.fieldname nicer than getter functions like fieldname(object) or get_fieldname(object) . Maybe have object.fieldname (or object$fieldname) being a call to getpublicfield (maybe with a better name) and object..fieldname being the actual getfield (private) could be a good option. In that way, types should define getpublicfield instead of getters, and trying to do object.fieldname should give an error id the field is private (It will be private if it doesn't have a definition for getpublicfield).

I added the decision label. This issue has been discussed in length and either it has to be done or not. When reading to #5848 it seemed that @JeffBezanson @StefanKarpinski and @stevengj want this. If yes then this issue should get a milestone so that it is not forgotten. Otherwise close. In any case I think this is a change that should be done pre 1.0.

@JeffBezanson and i were just discussing this yesterday. Tentative conclusions: (i) yes, we should have this; (ii) don't allow dot overloading for Module (which will be specially handled); (iii) don't supply any special syntax for Core.getfield (since there's no pressing need for an overloaded getfield to have the same name as a "real" field; the latter can just start with an underscore).

@stevengj: Sounds like a reasonable plan. Could you indicate if this will be restricted to single argument or if the multi argument version a.fieldname(b) should also be supported? This will draw a conclusion to the above discussion. Furthermore it would be great to put an appropriate milestone label to this (1.0?). Thanks!

Jeff and I didn't discuss the multi-arg case. My feeling is that we might as well support it, since you can simulate it anyway by returning a function from the no-arg case (but it isn't critical to do right away for the same reason).

I use a converter to cast values and validate data.
like this:

abstract AbstractAge{T}
abstract AbstractPerson
type PersonAge <: AbstractAge{AbstractPerson} 
    value::Int64
end

Base.convert(t::Type{AbstractAge{AbstractPerson}}, value::Int64) =  begin
  if value < 140 && value > 0
    PersonAge(value) 
  else
     throw(ErrorException("ValueError"))
  end
end

type Person <: AbstractPerson
  age::AbstractAge{AbstractPerson}
end 

a = Person(32)
a.age = 67

Here's a fun 3-line implementation of this:

diff --git a/base/boot.jl b/base/boot.jl
index cd3ae8b..a58bb7e 100644
--- a/base/boot.jl
+++ b/base/boot.jl
@@ -266,6 +266,9 @@ Void() = nothing

 (::Type{Tuple{}})() = ()

+struct Field{name} end
+(::Field{f})(x) where {f} = getfield(x, f)
+
 struct VecElement{T}
     value::T
     VecElement{T}(value::T) where {T} = new(value) # disable converting constructor in Core
diff --git a/src/julia-syntax.scm b/src/julia-syntax.scm
index b4cb4b5..59c9762 100644
--- a/src/julia-syntax.scm
+++ b/src/julia-syntax.scm
@@ -1685,7 +1685,7 @@
     (if (and (pair? e) (eq? (car e) '|.|))
         (let ((f (cadr e)) (x (caddr e)))
           (if (or (eq? (car x) 'quote) (eq? (car x) 'inert) (eq? (car x) '$))
-              `(call (core getfield) ,f ,x)
+              `(call (new (call (core apply_type) (core Field) ,x)) ,f)
               (make-fuse f (cdr x))))
         (if (and (pair? e) (eq? (car e) 'call) (dotop? (cadr e)))
             (make-fuse (undotop (cadr e)) (cddr e))

My thinking is that a.b should actually call a projection function Field{:b}() instead of getfield, so that you get functions like x->x.a already for free. That also allows getfield to always mean low-level field access.

The above implementation completely works but is pretty hard on the compiler (sysimg +5%, which is kind of a pleasant surprise really). So this will need some specialization heuristics and some early optimizations need to be updated, but then hopefully this will be viable.

I'm surprised tests could pass with that implementation. Doesn't this semantic make it impossible for codegen to optimize module references? It also seems like it'll make global variables much more expensive (speed and memory). It doesn't seem too likely for this to show up in the sysimg. Although the inference degradation seems like it should have made the sysimg smaller here, so 5% worse doesn't sound like a good start)

Yes, there is code in jl_resolve_globals to convert these to GlobalRefs which would need to be updated. Other than that, these should get inlined so it should be possible for codegen to handle them. In inference, we probably just need a special case similar to that for tuple getindex.

we probably just need a special case similar to that for tuple getindex

That special case hopes and assumes that no method will be defined that intersects with the builtin getindex method signature. I don't see how that case is especially applicable here.

I would add a method for ::Module and say you're not allowed to shadow it --- possibly enforced with an actual error.

@JeffBezanson Do you have a branch with those three lines? I tried to add them myself to a local build, but julia seems to have moved along by now and I couldn't get it to work.

I should say that if I had only one wish for a feature in julia 1.0, this would be it. It would solve two long standing problems I have in both Mimi and Query.

The case for Query I think is pretty generic. I believe one could write a version of NamedTuples with this that didn't have to generate new types in macros, and that would in turn enable me to write generated functions that return a NamedTuple with a set of fields that are computed in the generated function. I think that would enable me to write a type-stable version of blackrock/NamedTuples.jl#4, which is by far my biggest stumbling block in Query.jl right now.

Long story short, this would be super, super valuable for the whole data handling story in julia.

Is the syntax below available?

function (obj::MyType).plus(n::Int)
       return obj.val + n
end

Why in the world would you want that syntax?

Wanting to write obj.plus(n) instead of obj + n is some serious o.o. Stockholm syndrome.

Ok, the given example was a trivial one (and a bad one).
It was just an idea for using this syntax instead of overloading getfield and be able to use arguments.

I'm mostly concerned with the functionality. Having an abbreviated syntax for overloading getfield seems much less important and not worth getting bogged down arguing about. Furthermore, getfield and setfield! directly mirror getindex and setindex! and so are somewhat natural in Julia. Finally, extensive use of an OO style is not really a good fit for Julia's idiom, and I don't think we want to encourage it by an OO-like method-definition syntax (but see my comment above regarding arguments).

One thing that occurred to me was that the $ operator was deprecated. Now, this obviously makes doing something like $(x, sym::Symbol) = ... available already, but we could also entertain a fancier syntactic rewrite like:

x$y          => $(x, ::Type{Val{:y}})
x$z(args...) => $(x, ::Type{Val{:z}}, args...)

I think that covers most of the cases mentioned in this issue already without doing full-blown getfield overloading. Frankly, the . operator is pretty semantically saturated in Julia, so something like this feels easier to digest and is convenient enough to still be useful.

@quinnj, already proposed in #18696.

Since we already have . for field access, however, it seems inelegant to have two field-access-like operators, one overloadable and one not. And it would be a bit unnatural for inter-language calls to Python and other OO languages, which almost universally use ..

I don't see interoperability with other languages as a valid argument for introducing something like this. It's like saying, "Python code looks like this, so to be able to pretend to be Python we should do this too." I've yet to see an argument for this that will make Julia itself better and/or more consistent. It's already well-established that Julia does not provide the OOP-style x.f() syntax; allowing things like that is asking for inconsistency.

@stevengj, part of where I'm coming from though is the fact that x.f _isn't_ field access. There is no actual field member f. This whole issue is about overloading getfield and I think the main concerns are the potential confusion of whether x.f actually refers to a field or is doing something else under the hood.

The advantage of the syntactic rewrite I proposed is that it fulfills the language interop story without introducing additional confusion with getfield; that's what I was referring to when I mentioned . would be over-saturated. Would x$f really be that much more cumbersome? I just don't see why this _has_ to use the dot operator.

This whole issue is about overloading getfield and I think the main concerns are the potential confusion of whether x.f actually refers to a field or is doing something else under the hood.

I don't think there is any potential for confusion in @JeffBezanson's three line proposal: a.b is always lowered into a projection function call, and never a getfield call. And then there is a default/fallback method in base for the projection function that calls getfield. Users can add methods for their own types to that projection function if they want. getfield always means low level field access in that proposal, and users wouldn't add methods to getfield (I assume that is what you meant by "overloading getfield"). That seems very clean to me. Having said that, it is not clear to me whether that proposal is still on the table or not, there seem to be some compiler concerns that I don't understand :)

I'd really love to have this feature (I've always loved it in Scala). IMHO accessing fields directly comes very natural in Julia (compared to, e.g., the Java way of defensively defining getters and setters for everything). But without the ability to add "virtual" fields, it becomes very difficult to evolve/refactor structures without breaking lots of code.

I know this has been tagged v2.0, but I would like to bring this up again in hopes of reconsidering this for v1.0. This feature will be very valuable for package/library developers.

When building a package for users, overloading the attribute accessor will allow package maintainers to present a much cleaner user interface. I would argue that,

complex_data_structure.attribute

is easy to type than

get(complex_data_structure, :attribute)

where get is a function required to get data that is described by :attribute.

This could potentially also improve discoverability in the REPL, and would benefit language code and data exploration.

Additionally, a setter attribute intercepter would also be incredibly valuable. Consider the following example (already found in many of my scripts currently :)),

set(complex_data_structure, :attribute, modify_attribute(get(complex_data_structure, :attribute), additional_arguments))

If a attribute getter and setter were included as part of Julia, the above would become the following.

complex_data_structure.attribute = modify_attribute(complex_data_structure.attribute, additional_arguments)

This in my humble opinion is much more readable. (Sidenote: I could use a pipe composition to make it more readable, but the additional_arguments again would complicate it there again. And the argument here would still stand.)

Furthermore, constraining values provided by a user is really common use case, and this will be really great to have in v1.0. It will greatly improve user interfaces for multiple packages. Currently, to the best of my knowledge, there appears no way to do the following (can someone correct me if I'm wrong)?

module point

mutable struct Point
    x::Int
    y::Int
    function Point(x, y)
        if x < 0 || y < 0
            throw(error("Only non-negative values allowed"))
        end
        this = new(x, y)
    end
end

end
# point

p1 = point.Point(-1, 0)
# Only non-negative values allowed

# Stacktrace:
# [1] point.Point(::Int64, ::Int64) at ./In[30]:8
# [2] include_string(::String, ::String) at ./loading.jl:515

p1 = point.Point(0, 0);
p1.x = -1;
p1
# point.Point(-1, 0)

The constructor can restrict the domain inputs to a immutable structure, however a user may still enter values that are not valid. Attribute getters and setters would help here, because they would make the feedback on data validity here more immediate, and this will make for a much more cleaner interface as a user of the language.

This will also greatly benefit other packages such as PyCall.jl and DataFrames.jl to build arguably better and more intuitive interfaces for users. The authors of these packages have also expressed interest above in having this feature.

Thoughts? Reading through the thread, it appears that this is already pretty close to final. I'm curious what the authors thoughts are on this?

Implemented in a type stable optional macro here: https://github.com/bramtayl/DotOverloading.jl. I think it has potential for Base.

@bramtayl, I think having to put @overload_dots before a.b expressions defeats the point here.

I wasn't suggesting the macro be added to base; I was suggesting the strategy used by the macro be baked into the parser. However, it's a macro that can be run on whole files or even hacked into an IDE.

That would entail a.b being parsed to Expr(:., :a, :(Val{:b}()) and lowered to get_field_overloadable(a, Val{:b}())

@bramtayl, see Jeff's implementation above.

One problem with overloading getfield is that some libraries do not have a reasonable interface to the internals of a data structure. When all I want is to modify a data point in a structure so that I can run my code and make my report for tomorrow, I am not particularly looking forward to editing code in a library three levels deep into my dependency so that I am able to directly access the data point in a reasonable and fast manner. I want to feel like I have control over the data structures I am using.

Second point is that when I am using getfield and setfield I want a reasonable expectation that I am directly accessing the data structure instead of some huge function mechanism. Besides, the keyword struct is used to define a type, reminding you of C, and I feel like it gives you the expectation that getfield access should be direct.

So, I think that using the $ operator instead is a reasonable compromise since there is no expectation that access is direct like in getfield, it is already a special character in other contexts and hence it won't be too surprising when it is used in this manner, it will be easy to deprecate because few people use it and because it won't be an function anymore, and it is used similarly in R.

One problem with overloading getfield is that some libraries do not have a reasonable interface to the internals of a data structure.

@JeffBezanson's implementation from above does not overload getfield (or setfield).

I'd also like to plead for this to make it into 1.0. I often find myself torn between writing getters/settings to keep my code somewhat flexible (and then wondering how to name them), or "allowing/encouraging" the user of my code to use direct field access.

Defensively writing lot's of getters and setters up front doesn't seem very Julian to me - after all, Julia has no private/protected fields and so encourages the user to access fields directly. Then there's the question on how to name a lot of getter-/setter-functions without risking lot's of conflicts with other packages. And the alternative, adding methods for getfield and setfield! (or similar), dispatched on Val{:fieldname} or so, doesn't seem very user-friendly, either.

On the other hand, if direct field access basically locks all structs forever, it clearly shouldn't be encouraged - especially for long-lived code. @JeffBezanson's implementation would seem to be such a neat way out of this dilemma.

Right, @davidanthoff. I must be conflating overloading of getfield and overloading of the . field access syntax. I meant overloading of the field access syntax.

There are backwards compatible ways to add this feature, so it could be added in a 1.x release, but it's not happening in 1.0. We don't have a finished design or time to implement it even if we had one.

It would be supremely convenient to use dot-syntax on pointers to (C-) structs.

My preferred syntax would be to use dots for moving and converting pointers only, and using [] for deref.

So, struct somepair a::Int b::Int end, and p::Ptr{somepair}, then p.a is a pointer to the a field, and I write to it with p.a[] = 3, or read with x = p.a[].

Next we just need forward declarations of types, and possibly a way to import C headers, and then wrapping C becomes a breeze.

Ps, to clarify, something like:

function getfield(p::Ptr{T}, fn::Symbol) # dispatch on values of T and fieldname
ftype = fieldtype(T, fn)
offset = fieldoffset(T,fn)
return convert(Ptr{ftype}, p+offset)
end

getindex(p::Ptr{T}) where T = unsafe_load(p)
setindex!(p::Ptr{T}, v) where T = unsafe_store!(p, convert(T,v))

For bonus points, the types from julia's runtime lib could be exported, and pointer_from_objref could actually give us a nicely typed pointer for more convenient introspection.

I have a feeling that unions should kinda work automatically, just by having two fields with the same offset?

@chethega I think you're looking for https://github.com/JuliaLang/julia/pull/21912, which provides roughly the same feature, without the issues surrounding adoption of C's memory model (type-punning violations, out-of-bounds access, aliasing of interior pointers, etc, which limit the performance optimizations possible in that language).

Does constant propagation make this more doable?

Pls change milestone to 1.0! :)

@Liso77 What do you mean? This is already implemented on git master, as noted in the status when closing.

@nalimilan sorry if I understand it wrong! But I thought that as 1.x are labeled postponed things which are going to be solved after 1.0. And this is solved now...

Open source is a decentralized community. The milestone sets what should be complete by 1.0, but contributors and work on whatever they want. In this case someone wanted this in 1.0, so they contributed the code to get it there.

@Liso77 As I understand it, this will not be in v1.0. The Julia v1.0 feature freeze date was set to Dec 15th, but this issue was closed on Dec 17th so I think we can expect it in a 1.x release. Core devs can correct me if my interpretation is incorrect.

No, it is merged into master and will be in the next release.

:) Well! I just thought that it is good to label 1.0 what is going out in 1.0. If it is not wanted then sorry for disturbing! :)

I think the NEWS file is a better way to see what is going into 1.0.

The milestone was added to mean "can be implemented without breaking compatibility in the 1.x release series", but then since the code was ready it's been merged anyway before the 1.0 feature freeze. I've removed the milestone for clarity, but at this point everything which is merged in master will be in 1.0.

Thanks for the clarification! This is exciting! The NEWS file was also particularly enlightening.

Thanks for remove! I am also very glad it will come in 1.0! :)

I wonder if there's a way to support these new "dynamically-defined fields" in auto-complete, e.g. by allowing to overload fieldnames?. This could be very powerful, for interactive use, e.g. when dealing with DataFrames (assuming they'll will support df.column_name in the future) with many columns and/or long column names.

I guess at the moment the REPL (tab-expansion), IJulia, etc. look at the type definition, not the instance? But maybe that could be changed, for interactive use. It's probably impossible to support in an IDE like Juno, though, as instances are not available during coding.

@oschulz, you can already overload fieldnames:

julia> struct Foo; foo; end

julia> fieldnames(Foo)
1-element Array{Symbol,1}:
 :foo

julia> Base.fieldnames(::Type{Foo}) = [:bar, :baz]

julia> fieldnames(Foo)
2-element Array{Symbol,1}:
 :bar
 :baz

And it looks like fieldnames is what the REPL looks at:

julia> x = Foo(3)
Foo(3)

julia> x.ba<tab>
bar baz

@yurivish right - but is it "safe" to do so, currently? I'm not sure what else relies on fieldnames.

If not safe, it should be possible to define a complete_fieldnames(x) = fieldnames(x), use complete_fieldnames for the REPL completions, and overload that for custom completions.

Yes, this is why I was bringing this up - in case something needs to be changed/added in Base, so that REPL and friends can take advantage of it later. In view of the 0.7 feature freeze ...

This is why I'm glad we called the function getproperty; it helps clarify that it doesn't refer to the same thing as fieldnames. Interfering with fieldnames could definitely cause serious problems.

We might need a corresponding propertynames to give the valid property values. Of course, that may not be a well-defined concept, so perhaps we want something more specific to completion.

I had a feeling that this may require a bit of deeper analysis and input from the experts (thanks!) - should we open a new issue for this?