cc @nrc @jseyfried

I'd love for #[proc_macro_attribute] to be implemented soon. I already have a prototype and test usage that I banged out before realizing there's no compiler support yet :unamused: :

Prototype: https://github.com/abonander/anterofit/blob/proc_macro/macros/src/lib.rs
Example/Test: https://github.com/abonander/anterofit/blob/proc_macro/examples/post_service_proc_macro.rs

Tasks

(dtolnay edit: moved the checklist up to the OP)

cc @nrc @petrochenkov @durka @Ralith

@jseyfried I ran into an issue where if a legacy macro and an attribute with the same name are imported into the same scope, trying to use the attribute throws an error that macros cannot be used as attributes. Can we make this work so that both can be in the same scope and can be used as intended?

@abonander All macros (bang, attribute, and derive) share the same namespace, so we can't use two different macros with the same name in the same scope. However, we could improve that error message -- could you open an issue?

Sorry I’m late to the party. I’m happy with the direction to expose tokens rather than an AST, but I have some concerns about the specific TokenStream API proposed in the RFC:

pub enum TokenKind {
    Sequence(Delimiter, TokenStream),

    // The content of the comment can be found from the span.
    Comment(CommentKind),

    // `text` is the string contents, not including delimiters. It would be nice
    // to avoid an allocation in the common case that the string is in the
    // source code. We might be able to use `&'codemap str` or something.
    // `raw_markers` is for the count of `#`s if the string is a raw string. If
    // the string is not raw, then it will be `None`.
    String { text: Symbol, raw_markers: Option<usize>, kind: StringKind },

    // char literal, span includes the `'` delimiters.
    Char(char),

    // These tokens are treated specially since they are used for macro
    // expansion or delimiting items.
    Exclamation,  // `!`
    Dollar,       // `$`
    // Not actually sure if we need this or if semicolons can be treated like
    // other punctuation.
    Semicolon,    // `;`
    Eof,          // Do we need this?

    // Word is defined by Unicode Standard Annex 31 -
    // [Unicode Identifier and Pattern Syntax](http://unicode.org/reports/tr31/)
    Word(Symbol),
    Punctuation(char),
}

pub enum StringKind {
    Regular,
    Byte,
}

It’s not clear if this API was intended as a complete plan that was accepted when the RFC was merged, or just an example to be worked out later.

Generally, this seems far from the "normal" Rust syntax accepted by the compiler outside of macros. While some macros will want to parse some ad-hoc domain-specific language, others will want to parse "actual Rust" syntax and make sense of it.

1. (Minor) I don’t think Eof is necessary. An Iterator will presumably be used to, well, iterate over a TokenStream and Iterator::next already returns None to signal the end of iteration.

2. (Minor) I don’t think Exclamation, Dollar, or Semicolon are necessary. Matching on Punctuation('!') for example is not more difficult.

3. (Minor) As others have mentioned in the RFC PR, we might want to omit comments that are not doc-comments. (Any use case that wants to preserve comment likely wants to preserve whitespace too.)

4. As far as I can tell, what to do with multi-character operators (that probably should be a single token each) is still an open question. A possible solution is discussed in PR comments, but it looks like that didn’t make it into RFC text.

5. Number literals are missing. Are macros supposed to parse [Punct('1'), Punct('_'), Punct('2'), Punct('3'), Punct('4'), Punct('.'), Punct('5'), Punct('e'), Punct('6')] by themselves to evaluate a literal? They can’t even use str::parse::<f32> to do that, since the syntax it accepts is not the same as Rust literal syntax (which can have _ in the middle, for example).

   I imagine that there’s a stability concern here. Can we introduce new numeric types like u128 / i128 (and possibly in the future f128, u256, …) and their literals, without breaking changes to the tokens API? One way to make this possible might be:

   struct IntegerLiteral { negative: bool, decimal_digits: String, type_suffix: Option<String> }
   impl TryInto<u32> IntegerLiteral { type Err = OutOfRange; /* … */ }
   // Other impls for integer types supported in this compiler version
   
   // Something similarly for floats
   

   Or maybe something else. But I don’t think "pretend numbers don’t exist" is a good way to do it.

6. // Word is defined by Unicode Standard Annex 31 -

   This definition needs to be more precise than that. UAX 31 specifies a couple different variations of identifier syntax, and none of them is called "word". But choosing which exact variation we want is why non-ASCII identifiers are feature-gated at the moment.

   Instead, I think this should be defined as "whatever the current compiler accepts as an identifier or keyword" (which can change per #28979). Maybe with a pub fn is_identifier(&str) -> bool public API in libmacro.

7. Unicode strings and byte string literals share a single token variant, which I think is wrong as the memory representations of their values have different types (str vs [u8]). It’s also not clear if the text: Symbol component is intended to be a literal slice of the source code or the value after resolving backslash escapes. I think it should definitely be the latter. (For comparison, Char(char) has to be the latter since \u{A0} takes more than one char to represent literally.)

another way to write high level macros would be using lisp like macros, but this would need a s-expression representation for the whole rust ast.

@SimonSapin,

As others have mentioned in the RFC PR, we might want to omit comments that are not doc-comments. (Any use case that wants to preserve comment likely wants to preserve whitespace too.)

Please don't. I have a use-case where I want to use (though not preserve—they will be written into a separate compilation product instead) comments in the syntax.

Specifically, I want to create translation macros that would load translations of a string from a separate source file(s) and I would like to generate a list of strings to be translated as by-product in debug build. And there needs to be a way to include comments to be emitted into that list (rust-locale/rust-locale#19). So it makes sense to use comment syntax and the macro needs to see them.

I agree with the other points in that post.

@jan-hudec
Even if we didn't have a TokenKind::Comment, you could still use comments by looking at the contents of the spans between consecutive tokens.

I don think we shouldn't have TokenKind::Comment to encourage procedural macros to ignore comments so that users are free to add comments to macro invocations without worrying about changing semantics.

@jan-hudec Is there a reason attributes won't work with your solution?

@abonander, attributes absolutely don't make sense. Translatable strings act as literals, not as items. But extracting them during compilation would be just for convenience—it can always be done as separate parsing (and in fact, may end up being so, because I need to see _all_ of them in the crate and incremental compilation would break that).

I want to make a procedural macro that's based on serde's derive (and so calls the serde tokenstream functions directly) but there's no way to say I want to consume serde derive as a library rather than a procedural macro. This isn't exclusive to derive macros, I can see a similar thing being wanted for 'normal' procedural macros too.

My only solution right now appears to be forking serde_derive.

The the problem is this error message from rustc:

error: the `#[proc_macro_derive]` attribute is only usable with crates of the `proc-macro` crate type

It's easy to remove that and make it work, but there is also some complexity that I'm not sure how to resolve - a procedural macro crate could plausibly want to both use the proc-macro derive from another procedural macro crate, as well as calling the functions to generate the derive for a downstream user. What would that look like? Is there anything similar like this around at the moment, where a crate can be linked to in two different ways at the request of the consuming crate?

@aidanhs

a procedural macro crate could plausibly want to both use the proc-macro derive from another procedural macro crate, as well as calling the functions to generate the derive for a downstream user. What would that look like?

You can't access the functions (or anything else besides procedural macros) from a proc-macro crate. If you want to use TokenStream -> TokenStream functions and the corresponding procedural macros, you'll need to put the TokenStream -> TokenStream functions in a separate, non-proc-macro crate, and then also have a proc-macro crate that just delegates to those functions.

This RFC will be mostly implemented once #40939 lands.

Provide a way for proc_macro authors to create expansions that use items in a predetermined crate foo without requiring the macro user to include extern crate foo; at the crate root

Suppose that I want to present a single crate, that contains both non-macro items and a procedural macro that refers to said items. When #40939 lands, will this three-crate pattern be the idiomatic way to achieve this goal?

1. Put all the non-macro items in foo_runtime
2. Implement the procedural macro in foo_macros, referring to the symbols in foo_runtime as necessary
3. Add a final "façade" crate foo that pub uses the items from foo_runtime and foo_macros
   
   
   
   • This is the only crate that the user will import directly
   
   
   • This works because the hygiene system fixes the macros to point to the right crate
   
   

I ask because my use case involves importing two crates, and it would be great for usability if I could get away with just one.

@lfairy I think a "two-crate" pattern will be the idiomatic way:

1. Put all non-macro items in foo
2. Implement the procedural macro in foo_macros, referring to symbols in foo as necessary, e.g.

#[proc_macro]
fn m(_: TokenStream) -> TokenStream {
    quote! {
        extern crate foo; // due to hygiene, this is never a conflict error
        foo::f();
        // --- or just --- (if/when we get the sugar)
        $universe::foo::f();
    }
}

1. pub use items from foo_macros in foo.

This works because the hygiene system fixes the macros to point to the right crate

Reexporting a procedural macro in a different crate does not affect how names from the procedural macro resolve.

@jseyfried: Do you know whether this re-exportation trick also works with custom derives? Because these crates have exactly the same limitation of not being able to export any items.

@colin-kiegel
Custom derive crates are just proc macro crates that happen to only have #[proc_macro_derive]s.
With #[feature(proc_macro)], you can re-export custom derives in ordinary crates, just like you can re-export other proc macros.

@jseyfried I'm aware of the situation as it is at the moment, I posed the question because I don't think it's ideal and hoped to have a discussion about it. In the situation you describe, 'delegating'/reusing to the procedural macros of another crate becomes a matter of convincing the macro author (in this case, serde) to split their procedural macros into two crates. If I can call procedural macros like normal functions, the upstream crate author doesn't even need to know I'm using their crate.

That said, I recognise the compatibility hazard - the exact tokentree generated by a macro becomes part of the stable interface, so if serde changes how they generate Derive in a patch version and I've written a fragile macro, my macro will be broken for every single new user of my crate (as opposed to a fragile macro in the current case, where in the worst case it'll only work for specific inputs, but consistently).

@jseyfried

Does this pull foo from the current cargo deplist? That sounds bad (i.e. will it do anything particularly stupid if there are 2 crates named foo linked into the current binary?).

@aidanhs That would be a major language change/addition that would warrant its own RFC.

@arielb1

Does this pull foo from the current cargo deplist? That sounds bad

Yeah -- sadly, quoted extern crate names aren't hygienic, i.e. the resolution depends on which crate names happen to be in scope where the procedural macro is used. We can mitigate this using the re-export trick (i.e. re-exporting foo_macros in foo so that we know foo will be in scope), but that doesn't protect against ambiguity errors when there are two crates named foo.

I think the best solution here is to add phase 1 (i.e. target w.r.t. host vs target) dependencies to the Cargo.toml for proc-macro crates via a --target-extern command line argument. This would allow us to explicitly list the extern crate names in scope inside quote!.

@jseyfried

The idea is that a proc-macro crate would have a dependency in its "target" metadata, right?

@arielb1 Yeah, exactly.

This RFC will be mostly implemented once #40939 lands.

@jseyfried As in, ready to be stabilized when that PR lands? If not, what would remain blocking stabilization? I just don't want this to be yet another feature where it feels like we get 95% of the way towards implementing and people get all excited, and then things peter off anticlimactically.

As in, ready to be stabilized when that PR lands?

No, we want to get some experience with the API before stabilizing and perhaps future proof hygiene for extern crate names (i.e. address this issue that @arielb1 pointed out).

We will probably want to make breaking changes to this API; @eddyb has proposed/considered generalizing OpKind to all token trees. Also, we might change how we handle doc comments, floating point literals, etc. Overall, the API in this PR isn't mature enough to consider stabilizing yet.

@bstrie sadly the RFC to fast track proc macro stabilisation (with a limited api where e.g. token streams are only accessible through their string representation) like the derive macro stabilisation has failed: https://github.com/rust-lang/rfcs/pull/1913

@est31 Postponed, more like -- after a little experience with this API we might agree on a subset that we can agree to fast-track to stable.

The String-based API interacts badly with declarative macros 2.0 and is already limiting today, even without macros 2.0 and just with #[derive]s. We want to avoid proliferation of the String based API as much as possible to avoid issues as people migrate to macros 2.0.

I've opened an issue for #[proc_macro_attribute] seemingly not getting expanded on trait methods (maybe trait items in general?)

Since this is now the tracking issue for the proc_macro crate and its new APIs I thought I'd write down some thoughts as well. I've published a crate called proc-macro2 which is intended to be the exact same as the proc_macro crate in-tree except that it provides the ability to compile on stable Rust. It then also has the ability to use a feature to compile on nightly Rust to get the benefit of better span information. That library is intended to become the foundation for other libraries like syn, and in the development of syn we found a few shortcomings we may wish to address in proc_macro directly:

• There's no Literal constructor for a few sorts of literals. This is worked around via stringification followed by parsing, but it'd be great to be able to construct these directly without having to go through the string API.
  
  
  
  • Raw strings - r###" foo "###
  
  
  • Raw byte strings - rb#" foo "#
  
  
  • Byte literals - b'x'
  
  
  • Doc comments - these are currently represented as the Literal token.
  
  
• There's no way to inspect a Literal and extract its value. Right now we're relying on literalext crate to to_string a literal and re-parse it, but this information in theory is already stored in the Literal and it'd be nice to be able to access it.
• The mapping of tokens in some cases can be construed as a bit odd. Namely right now doc comments map to the Literal type.

I believe all other concerns starting here have since been addressed.

I encountered a breakage when testing with #![feature(proc_macro)] that affects custom derives that have #[proc_macro_derive(foo, attributes(foo))]. That is, a custom derive which has the name of an attribute that is the same as the custom derive. One such crate is mine - derive-error-chain, which has #[derive(error_chain)] #[error_chain(...)] struct ErrorKind { ... }. Another is derive-new, which has #[derive(new)] #[new] struct S;. I don't know if there are others.

For code like this, the compiler complains at the second attribute that "foo" is a derive mode. Is this intentional or can it be fixed? If intentional I need to prepare to rename my custom derive to ErrorChain or something.

@Arnavion
This was intentional in general -- since proc_macro_attributes must be expanded before derives, if new were proc_macro_attribute then the expansion would be ambiguous. It would be possible to specifically allow new to be a proc_macro_derive, but I'm not sure it's worth it (also could be a future-compatibility hazard).

This was intentional in general -- since proc_macro_attributes must be expanded before derives, if new were proc_macro_attribute then the expansion would be ambiguous.

Okay, I'll rename #[derive(error_chain)] to #[derive(ErrorChain)].

It would be possible to specifically allow new to be a proc_macro_derive, but I'm not sure it's worth it (also could be a future-compatibility hazard).

Sure, I wasn't asking for new to be special-cased. It was just an example from one of the two proc_macro_derives I know about that are broken by this.

@Arnavion Sorry, my last comment wasn't the clearest -- I didn't mean special-case new specifically but to allow #[derive(some_macro)] #[some_attr] struct S; when some_attr resolves to a proc_macro_derive. When some_attr resolves to a proc_macro_attribute, this would need to be an ambiguity error; today, it is an ambiguity error if some_attr resolves to any macro.

Yes, I got it.

(I hope this is the right place for a question like this.)

What's the status of this?

• [ ] Provide a way for proc_macro authors to create expansions that use items in a predetermined crate foo without requiring the macro user to include extern crate foo; at the crate root (PR #40939).

The PR has landed, but the box is still not checked. @jseyfried mentioned something here and it kinda seems to work. However it doesn't seem to work with use at all:

let call_site_self = TokenTree {
    kind: TokenNode::Term(Term::intern("self")),
    span: Span::call_site(),
};
quote! {
    extern crate foo; // due to hygiene, this is never a conflict error

    // Neither of those works    
    use foo::f;
    use self::foo::f;
    use $call_site_self::foo:f;
}

Am I missing something? What is the idiomatic way to use symbols from an extern crate imported in the macro?

You can't use use see https://github.com/rust-lang/rfcs/issues/959. But for a macro it's not really a disadvantage to use the fully qualified path everytime. (Except for traits, I think)

@parched Thanks for linking this other issue. My use case was the following:

In my macro I want to let the user write something similar to a match-matcher. Specifically, the user writes a Term and this can either be a variant of an enum or a simple variable name which binds the match value. To write some pseudo code with macro_rules! syntax:

macro_rules foo {
    ($matcher:ident) => {
        match something() {
            $matcher => {}
            _ => {}
        }
    }
}

Now I want the user to be able to just specify the variant name without the enum name. Thus I would insert a use my_crate::AnEnum::*; statement in the generated code. But since this is not possible (right now), I need to check for myself whether or not the $matcher is a variant of the enum or not.

I hope my explanation is understandable. I just wanted to give another use case for use in macro-generated code.

@LukasKalbertodt Could you just use my_crate::AnEnum::$matcher => {} in the match?
Nevermind, I your the issue -- I believe we'll need https://github.com/rust-lang/rfcs/issues/959 for that.

@jseyfried No: $matcher can either be a variant name (in which case your solution would work) or a simple variable name like in match x { simple_var_name => {} }. In the latter case it wouldn't work AFAICT. (btw, I just wanted to mention another use case to show that using use is important)

@jseyfried

This was intentional in general -- since proc_macro_attributes must be expanded before derives, if new were proc_macro_attribute then the expansion would be ambiguous.

Okay, I'll rename #[derive(error_chain)] to #[derive(ErrorChain)].

It seems attributes of custom derives also conflict with macro_rules macros, instead of overriding them like custom derives do based on import order. That is, this code compiles:

#![feature(proc_macro)]
#[macro_use] extern crate error_chain; // macro_rules! error_chain
#[macro_use] extern crate derive_error_chain; // #[proc_macro_derive(error_chain, attributes(error_chain))]

#[derive(error_chain)] // No error. Resolves to custom derive
enum ErrorKind {
    /*#[error_chain]*/ // (1) As discussed above, can't use this any more since it conflicts with the name of the custom derive
    Foo,
}

This matches the behavior of current stable Rust, with the exception that (1) does work in stable. I've even explicitly documented that users wishing to use #[macro_use] with the error-chain crate will need to import it before importing derive-error-chain.

But even if I rename the custom derive to ErrorChain to make (1) work with the proc_macro feature (which is already one breaking change for stable code):

#![feature(proc_macro)]
#[macro_use] extern crate error_chain; // macro_rules! error_chain
#[macro_use] extern crate derive_error_chain; // #[proc_macro_derive(ErrorChain, attributes(error_chain))]

#[derive(ErrorChain)] // Unique name, so no error
enum ErrorKind {
    #[error_chain] // (2)
    Foo,
}

it still doesn't compile - the attribute at (2) yields the error: macro `error_chain` may not be used in attributes because the macro_rules macro apparently conflicts with the attribute registered by the custom derive instead of being overridden like in the first case.

So I have to rename both the custom derive and its attribute. The attribute gets used a lot more (one on each variant of the enum) than the custom derive (one on each enum), so this is a bigger breaking change than I expected. I do understand that this is a tricky situation of my own construction (reusing the name of the macro_rules macro for a custom derive and its attribute), but this is also code that has been compiling in stable since custom derives were stabilized, so I had no reason to think it would be a problem six months later.

Can it perhaps be made so that attributes of custom derives override macro_rules macros just like custom derives themselves override macro_rules macros? Actually I don't see how there could be any ambiguity between them, but I assume it's the same reason as when a macro_rules macro is imported after a custom derive of the same name - that all macros are put in the same namespace without considering what kind of macro they are.

Is there some less formal "place" to talk about proc macros? Like a #rust-proc-macro IRC channel? I'd love to ask small questions about the feature from time to time, but it just feels wrong to spam this thread :see_no_evil: And in the #rust channel, most people haven't worked with proc-macros and especially the new proc_macro API (since it's unstable and all). So: any idea where to discuss this topic?

@LukasKalbertodt #rust-internals, maybe, or just start a new thread on /r/rust.

TokenStream::from_str panics when used outside of a procedural macro (for example in a build script):

thread 'main' panicked at 'proc_macro::__internal::with_sess() called before set_parse_sess()!', /checkout/src/libproc_macro/lib.rs:758:8

Would it be possible/desirable to replace this panic with implicitly creating a dummy "session"? Or perhaps add a public API (with a path to stabilization) to create one?

Has anyone looked at literature on macros from other systems? I would like to hear people's thoughts on this. I'll speak about Scheme here, since that's what I'm most familiar with.

I am personally working on implementing syntax-rules for R7RS Scheme on my own project, and I have found that syntax-case can form the basis for supporting both unhygienic and hygienic macro systems (defmacro and syntax-rules). GNU Guile does this. syntax-case also has support for fenders which can perform additional predicate validation on syntax object lists (or, something among the lines of TokenStream in Scheme). I can see that Mark is being worked on, and it looks like it's inspired by Bindings as Sets of Scopes.

Also, should we also discuss whether arbitrary computation at compile-time should be supported? Racket actually takes an entire "phase" approach to things, it appears like, with begin-for-syntax allowing for definitions and computation (?) at the compile-time level during macro expansion..

Control over hygiene is very possible with (datum->syntax <thing-to-copy-scope-from> <thing-to-apply-scope-to>) in Scheme, allowing you to escape the scope of a macro and instead take on a scope of an object outside of the immediate scope.

Take this example from The Scheme Programming Language, 3rd ed. by R. Kent Dybvig (Chez Scheme, now at Cisco Systems): http://www.scheme.com/tspl3/syntax.html. The example shows (include "filename.scm") as a syntax-case macro, and allowing the interpreter to use a macro to setup the runtime to read from a file and continue evaluation. The deeper question here is whether we want a macro-macro system to allow such things to happen at macro-expansion time, and trigger compile-time computations such as triggering a file import (although, this appears to occur in the direct compiler functionality, so maybe we don't want to do this).

What should the limits of macros be? I would imagine that Rust, wanting to cut down on its compilation time, wants to restrict compile-time evaluation (and especially avoid infinite loops). Racket has taken the "tower of preparers and expanders" approach with phases as referenced in Lisp in Small Pieces. Do we want to allow things like allow access to a compile-time API to perform file I/O and limited recursive computation? Should we allow things like have procedural macros be able to turn CSV spreadsheet specs into switch statements?

I'd love to hear about other systems! I hear Template Haskell has an interesting approach with well-defined types to represent their AST, and laziness in Haskell can replace many uses of macros for control structures.

Sorry if I'm stepping out of line.

What should the limits of macros be?

For procedural macros, discussed in this issue, none. A procedural macro is a compiler extension. It can take a bit of C++ code, run it through clang and add the resulting object to the compilation. It can take some SQL, query the database to find corresponding result type and generate appropriate result set. Those are actual use-cases people want to do!

Note, that Rust has another macro system. It's update was approved as RFC 1584 and implementation is tracked by https://github.com/rust-lang/rust/issues/39412.

@VermillionAzure, from quick look at the Scheme forms you referenced:

The macro_rules, and their update per RFC 1584, is similar to syntax-rules. If you have suggestions for enhancing those, https://github.com/rust-lang/rust/issues/39412 is probably the best place to discuss that.

The proc-macros, about which this issue is, are like the general form of define-syntax. And this RFC (1566) very intentionally does not define anything like the syntax-case. Only an interface to call a function to transform the token stream.

The interface is defined in such a way that something like syntax-case can be implemented in a separate crate (library) and the intention is to do it that way. If you are so inclined, feel free to play around. Both any prototype and report on how easy or hard to use the API is will certainly be welcome.

I think the idea was, defining macros like functions, like in lisp, but having a macro, which returns the macro, that macro_rules! defines.

So following would be equivalent:

    macro_rules! foo {/*define macro here*/}

#[proc_macro]
pub fn foo(tokens: TokenStream) -> TokenStream {
    macro_case! tokens {/*define macro here*/} //takes `tokens` as first argument, returns a `TokenStream`
}

That's how syntax-rules and syntax-case seem to work in scheme.

@VermillionAzure
Is this, what you would like?

@porky11 No, it doesn't seem so. I just wanted to see if Scheme macros would be a relevant idea to add to the discussion -- it is obvious that since procedural macros are intended to be much more powerful than the syntax-case macro system in Scheme that it is trivial to implement all of the macro systems in terms of the arbitrary power provided here.

@jan-hudec Is it wise to allow any arbitrary computation as a compiler extension without any sort of security guarantee? I am absolutely floored by the idea that procedural macros are going to be so powerful here, but would any potential users of Rust consider this a downside to using packages? I'm not a security expert by any means, but couldn't vulnerabilities in any libraries that are used within compiler extensions could easily be used to maliciously turn the Rust compiler into an attack vector? Additionally, if any bugs occur in libraries used in procedural macros (e.g. segfault triggered by bad C library code), does this mean that the segfault would trickle up and make the compiler fail without proper error messaging?

Would there be a way to encapsulate errors that occur in procedural macros in a way that it would not affect any others parts of the compiler?

Another idea: when do procedural macros execute? If procedural macros can interact with code that has side effects that could be relevant (e.g. communicating with a stateful external server, mutating an external SQL database, getting a security key to log to an external system), then doesn't that mean that the order in which procedural macros are triggered by the compilation process is important?

@VermillionAzure Cargo packages can already have build scripts that execute arbitrary code at compile-time, so procedural macros don’t make things worse on that front: you already need to trust your dependencies. (This is made somewhat easier by crates.io being immutable/append-only, and dependencies not updating automatically if you have a Cargo.lock file: you only need to trust specific versions.) And even if build scripts did not exist, your dependencies can still by nature execute arbitrary code at run-time. Is compile-time much worse?

This discussion makes me think of a related, but different problem.

Suppose that a crate defines two proc macros: foo!() writes a temporary file, and bar!() reads that same file. A consumer of this crate invokes both foo!() and bar!() in the same module. Then, whether compilation succeeds or not will depend on which of foo!() or bar!() is expanded first. This ordering is implementation-defined, and if enough people write code like this, may become de-facto standard.

I'm not sure how much of an issue this is though. Just concerned about whether this will lead to a repeat of the struct field ordering shenanigans.

@SimonSapin

While I agree with your position, I should point out that there is one significant difference between compile-time execution and runtime execution:

It provides a different threat model since things tend to be compiled once and then deployed to many machines. (eg. Exploiting maintainer inattention plus a sandboxing shortcoming on a Linux distro build cluster.)

@lfairy Yes, this is the exact issue that Racket had over a decade ago back in 2002. Matthew Flatt, the highest contributor to Racket, created a paper called "Composable and Compilable Macros: You Want it When?. R. Kent Dybvig, who worked on Chez Scheme, also wrote a paper on phases in evaluation for library/modules in "Implicit Phasing in R6RS Libraries".

@SimonSapin Compile-time can potentially be much worse. If your compiler crashes randomly or performs malicious behavior that is triggered by the compiler, then I would bet that someone would end up writing a huge Reddit post that would be titled "Rust's Modules Are Inherently Unsafe" or something like that.

@VermillionAzure, I didn't read the articles very carefully, but I don't think they are relevant for the discussion, since the issues faced by Rust are very different from the issues faced by Scheme.

In Scheme one library can provide both functions and macros, so the compiler must correctly sort out which functions it needs compile-time and which it needs run-time. However, in Rust a crate either provides procedural macros, or runtime functions, so this division is (for the time being) obvious.

(Note that a crate providing runtime functions can also provide rule-based (hygienic) macros, but those are separate mechanism in Rust)

The problem @lfairy is talking about is one of ordering of the execution of the expander functions. In Rust, the compilation might be parallel for separate files and it might be incremental, so the order of execution of the expanders is undefined. But does either article actually address that? I didn't see it.

@jan-hudec Yeah, I guess you're right. But the order of evaluation will definitely matter if side-effects are allowed at compile-time, unless you can guarantee a certain module does not produce side-effects. Is a module "type-able?"

I think that a procedural macro “probably shouldn’t” have side effects because some details (see below) cannot be relied on, but we likely won’t have a type-system-like mechanism in the language to force them to be purely functional.

These details include ordering of execution and concurrency compared to other proc macros, and whether a proc-macro is re-executed in an incremental build. For the latter we may want to add something to declare dependencies similar to rerun-if-changed in build scripts. And like build script these declarations might be incomplete or buggy. We can’t statically prevent all bugs.

I think we should avoid guaranteeing anything about side effects (i.e. proc macro crates are not allowed to rely on side effects working, nor are they allowed to rely on being retriggered for anything other than a code change in the module in which they are applied (so, no global state).

We can later relax this requirement with a way to specify rerun-if-changed and other things.

(I've got a half-baked general build script/proc macro security proposal that will help here somewhat, but I haven't really penned it yet)

IMO proc macros / custom derive should be put into a sandboxed environment without any I/O or other connection to the outside and be evaluated by miri, maybe with a cranelift JIT.

@est31 that's a nice idea but stuff like diesel infer_schema! already
exists, bindgen needs to read files and run programs, even include! and
env! use I/O.

On Nov 9, 2017 06:19, "est31" notifications@github.com wrote:

IMO proc macros / custom derive should be put into a sandboxed environment
without any I/O or other connection to the outside and be evaluated by
miri, maybe with a cranelift JIT.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rust-lang/rust/issues/38356#issuecomment-343124957,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAC3n5VOPdKBsu81Sp3tp2XlIQ05L865ks5s0t_PgaJpZM4LMWlc
.

It looks like #40939 and #44528 have already merged... @jseyfried could you update the checklist?

@mark-i-m updated.

Can procedural macros generate procedural macros?

@VermillionAzure I haven’t tried it, but I don’t see a reason why they couldn’t. If course, like proc macros generating "run-time" code, they’d need to be in crates separate from those where they’re used.

In syn we ran into a limitation of proc_macro::TokenNode today -- block delimiters { ... } are associated with only one Span in the input TokenStream so there is no way to trigger an error that points to only the closing }. Rustc does not seem to have this limitation.

mod m {
    type T =
}

error: expected type, found `}`
 --> src/main.rs:3:1
  |
3 | }
  | ^

The best things we can do in a proc macro are point to the last token inside the block, point to the entire block, or point to the next token after the block, none of which is really what you want for an error like above.

A general solution would be to have Span::start and Span::end return a 1-character Span instead of a LineColumn as they do currently, then expose a way to go from Span to first line/column of span.

sp.begin().line // before
sp.line() // after

sp.end().line // before
sp.end().line() // after

sp.end() // after, not possible before

Mentioning @abonander who added that API in #43604.

have Span::start and Span::end return a 1-character Span instead of a LineColumn as they do currently, then expose a way to go from Span to first line/column of span.

That would make Span adopt the special behaviour that delimitered lists need, but would be wrong for all other spans. Generally it is not correct. Consider getting a span for an expression like foo(hi) by joining stuff. Now you want to point to the foo and take sp.begin(), but sp.begin() only points to the first character of foo.

I think better solutions would be add two spans to proc_macro::TokenNode::Group or to allow creation of arbitrary spans.

Span::begin/end return types may have to change though: https://github.com/rust-lang/rust/pull/43604#issuecomment-327643229

I am trying to get Span::def_site() to resolve things against what is in scope in my procedural macro. Am I misunderstanding how this is supposed to work?

Almost the same code works if I use Span::call_site() and have MySend defined in the main.rs, which is as I would expect. Haven't been able to get it working with def_site() though.

#![feature(proc_macro)]

extern crate proc_macro;

use std::marker::Send as MySend;
use proc_macro::{TokenStream, TokenTree, TokenNode, Term, Delimiter, Span};

#[proc_macro]
pub fn impl_mysend_for(tokens: TokenStream) -> TokenStream {
    let span = Span::def_site();
    let ident = tokens.into_iter().next().unwrap();
    vec![
        TokenTree { span, kind: TokenNode::Term(Term::intern("unsafe")) },
        TokenTree { span, kind: TokenNode::Term(Term::intern("impl")) },
        TokenTree { span, kind: TokenNode::Term(Term::intern("MySend")) },
        TokenTree { span, kind: TokenNode::Term(Term::intern("for")) },
        ident,
        TokenTree { span, kind: TokenNode::Group(Delimiter::Brace, TokenStream::empty()) }
    ].into_iter().collect()
}

#![feature(proc_macro)]

extern crate mac;

struct S;
mac::impl_mysend_for!(S);

fn main() {}

error[E0405]: cannot find trait `MySend` in this scope
 --> src/main.rs:6:1
  |
6 | mac::impl_mysend_for!(S);
  | ^^^^^^^^^^^^^^^^^^^^^^^^^ did you mean `Send`?

Tracking this on the Syn side: https://github.com/dtolnay/syn/issues/290.

@dtolnay
The issue here is that MySend is imported in phase 0 (i.e. for the host architecture when cross compiling), so it isn't available in phase 1 (i.e. when compiling for the target architecture).

The solution here is to allow proc-macro crates to have phase 1 (target architecture) dependencies so that we can import phase 1 items into scope.

Today, a workaround is to return:

quote! { // n.b. non-interpolated tokens from `quote!` have `Span::def_site()`
    mod dummy {
        extern crate std;
        use self::std::marker::Send as MySend;
        unsafe impl MySend for $ident {} // this line is equivalent to what you have above
    }
} 

You could also construct this manually, I'm just using quote! for convenience.

Due to hygiene, dummy/std/MySend will never collide with anything else in scope, so e.g. it is safe to use this macro more than once in the same module, it's safe for ident to be "MySend", etc.

This issue, as well as the need for and solution to the mod dummy, is described in more detail in https://github.com/rust-lang/rust/issues/45934#issuecomment-344497531.

Sadly, until phase 1 dependencies get implemented, this will be unergonomic.

Thanks @jseyfried! That works. Some followup questions:

• In the code in my previous comment, if I change impl_mysend_for to generate an impl for Send instead of MySend then everything compiles. What Send is this resolving to and why does it not hit the phase 0 vs phase 1 distinction? Is this working intentionally or by accident?

• What else is in scope that can be used by my def_site() tokens, like Send?

• If MySend needs to come from a library (like imagine we are deriving serde::Serialize) then the end user still needs serde in their Cargo.toml, even if we don't need them to write extern crate serde. Would it be possible to make extern crate with a def_site() ident resolve against the procedural macro's Cargo.toml, and extern crate with a call_site() ident resolve against the downstream Cargo.toml?

For the extern crates, I assume the crates would need to be made explicitly available to phase 1 by the proc macro.

#[proc_macro_derive(Serialize, attributes(serde), crates(serde))]

@dtolnay

In the code in my previous comment, if I change impl_mysend_for to generate an impl for Send instead of MySend then everything compiles. What Send is this resolving to and why does it not hit the phase 0 vs phase 1 distinction? Is this working intentionally or by accident?

Good question. Right now, the prelude is in scope at the definition site (unless the proc-macro crate is #![no_implicit_prelude]) and this is an accident (in some sense) due to the phase 0 vs phase 1 distinction as you point out.

However, for ergonomics I think that phase 1 should implicitly contain std at the proc-macro root (so that you can always quote!(use std::...);) and the prelude for convenience/ergonomics and since these are already implicit in phase 0. There will be a PR to add std at the root in phase 1 soon.

What else is in scope that can be used by my def_site() tokens, like Send?

Besides the prelude and (soon) std as discussed above, the only other things in scope in phase 1 are the proc-macros themselves (not the proc-macro functions, which are phase 0).

For example,

#[proc_macro]
fn f(input: TokenStream) -> TokenStream { ... }

#[proc_macro]
fn g(_input: TokenStream) -> TokenStream {
    quote! {
        f!(); ::f!(); // These both resolve to the above proc macro
        f(); // This doesn't resolve since the function is in phase 0
    }
}

Would it be possible to make extern crate with a def_site() ident resolve against the procedural macro's Cargo.toml, and extern crate with a call_site() ident resolve against the downstream Cargo.toml?

Yes, except I believe an extern crate with Span::def_site() should resolve against the phase 1 (target) dependencies of the procedural macro's Cargo.toml -- today's phase 0 dependencies are linked to libraries compiled for the host platform. Since phase 1 dependencies don't exist yet, the extern crate's name resolves unhygienically, which is annoying as you pointed out.

Once we have phase 1 dependencies, we won't need to quote extern crates in each expansion to begin with, so this will be less of an issue. However, we should still fix it -- the current plan is to try resolving against the proc-macro crate's target dependencies first and then fall back to the unhygienic resolution with a low priority warning cycle to avoid churn.

For the extern crates, I assume the crates would need to be made explicitly available to phase 1 by the proc macro.
#[proc_macro_derive(Serialize, attributes(serde), crates(serde))]

Interesting, we could implement it this way.

I was thinking instead that the phase 1 crate would be declared at the proc-macro crate root (e.g. #[phase(1)] extern crate foo;) so that it would be automatically available in all proc macros (e.g. quote!(use foo::bar);). Since extern crate is on its way out anyway, we could avoid declaring the phase 1 crates altogether -- all target dependencies from the Cargo.toml would automatically be in scope at the proc-macro crate root in phase 1.

In my code I found that I seem to be using spans for two purposes: to control name resolution, and to control error messages. Are these two inseparably linked, or would it be possible to make a span that mixes the name resolution aspect of one span with the line/col error message locations of a different span? I expect this to be a common need.

To be more specific, I have a trait brought into scope within the def_site of my custom derive and I want to invoke trait methods on the fields of the user's struct. If the field type does not implement the right trait, I want the error message to underline the corresponding struct field.

I can generate the method call with a def_site span which compiles and runs but error messages sadly always point to the derive attribute like we saw with Macros 1.1.

  |
4 | #[derive(HeapSize)]
  |          ^^^^^^^^

Or I can generate the method call with the same span as the struct field's ident or type, which shows the right underlines but fails to resolve to the trait in scope at my def_site.

  |
7 |     bad: std::thread::Thread,
  |     ^^^^^^^^^^^^^^^^^^^^^^^^

How can I resolve correctly and show errors the way I want?

@dtolnay That’s an excellent point, thanks.

I think that the proper way to fix https://github.com/rust-lang/rust/issues/46489 might be to make it so that the generated #[derive(…)] tokens have name-resolution-spans in the same scope as the type definition, and error-messages-spans at the quote! {} macro invocation that created them.

What's the story with hygiene currently? I have function like procedural macro that used to work (4 months ago) but as of rustc 1.24.0-nightly (b65f0bedd 2018-01-01) it complains the argument can not be found in scope.

Sorry, should have searched the issue tracker first, seems like I've just hit https://github.com/rust-lang/rust/issues/46489.

I filed #47311 which I believe currently blocks a correct implementation of derive(Deserialize). Procedural macros cannot construct a struct that has private fields.

Filed another, #47312 in which access to an unnamed tuple struct field like self.0 has different requirements on the span of the . token than access to a named struct field like self.x.

47311 and #47312 are fixed in #48082 and #48083, respectively.

I have the above two PRs waiting for review/comment.

Looks like https://github.com/rust-lang/rust/pull/41029 is done now?

That PR was abandoned but revived and continues to be worked on in #48465. Currently waiting on Crater.

@petrochenkov @nrc

Looking at syntax::ext::expand, it appears that proc_macro_attributes are currently not being processed in several contexts (non-exhaustive):

• on blocks (fold_block() is a noop)
• on statements/expressions (#41475, #43988)
• inside extern {} blocks (#48747)

RFC 1566 doesn't list specific AST node kinds that attribute macros can be applied to, suggesting they should be applicable to just about anything. But that could get a bit ridiculous so we should clearly establish what need to be processed but aren't, and where attributes should never be allowed but currently might be (#43988)

@abonander the intention is that proc macro attributes can be used anywhere a regular attribute can be and nowhere else, which should I think cover all of the above (although some are not stable, and if we stabilise proc macros, we should be careful to only stabilise uses which are stable for other attributes).

@nrc is there anywhere that enumerates those locations because the reference only says that attributes may be applied to any item. However, I'm almost sure that lint attributes can be applied to blocks and statements as well.

@nrc is there anywhere that enumerates those locations because the reference only says that attributes may be applied to any item. However, I'm almost sure that lint attributes can be applied to blocks and statements as well.

There is not afaik - there is an accepted RFC and an unstable feature flag for attributes on any expression, but I think we have only stabilised on statements and blocks. The reference is out of data.

This issue:

Stability check (proc-)macros (issue #34079).

is now closed WRT proc-macros. My PR just didn't add stability-checking for Macros 2.0 macros which is why the issue is still open (though it should probably just be a new issue instead).

@rfcbot fcp merge

I'd like to propose a subset of macros 2.0 story be stabilized as macros 1.2 for the Rust 1.28 release. Rust 1.28 enters nightly on May 10, 2018 (~2.5 weeks from this writing) and will become stable on August 2, 2018. I think that FCP may finish before the May 10 cutoff for 1.27 entering beta, but I'd like to hold off any stabilizations here until after that cutoff has happened and delay this to the 1.28 release.

This has been discussed on internals recently along with a number of issues registered by @petrochenkov and which should be fixed now (but not quite yet released in nightly). I think it'd be helpful to recap here so this is concretely the subset which would be stabilized.

Remember though that this is a subset. Functionality missing here does not mean it will never be stabilized or removed from the compiler. Rather this functionality will remain unstable after this proposed pass of stabilization to be stabilized at a later date.

use some_proc_macro_crate::bar;

#[bar]
fn baz() {}

use some_proc_macro_crate::bar;
bar!();

pub use some_proc_macro_crate::bar; // reexport an attribute or macro

extern crate proc_macro;
use proc_macro::TokenStream;

/// Invoked as `foo!()`
///
/// When invoked as `foo!(a b ( c ))` then the `TokenStream`
/// here will be `a b ( c )`.
///
/// The invocation is replaced with the `TokenStream` returned
#[proc_macro]
pub fn foo(a: TokenStream) -> TokenStream {
    // ...
}

/// Invoked as `#[bar]`
///
/// The first argument, `attr`, is the token stream inside of the attribute
/// itself. The second argument, `item`, is the token stream corresponding to
/// the item the attribute is attached to.
///
/// An attribute of the form `#[bar ( a b [ c ] )]` will have the `attr`
/// argument look like `a b [ c ]`. Note the lack of delimiters passed to
/// `attr`! An API may later be added to learn what delimiter a macro was
/// invoked with.
///
/// The `item` here is a tokenified version of the original item.
///
/// The return value here will contain all non-expanded attributes as well for
/// this attribute to inspect. The return value replaces the original item.
#[proc_macro]
pub fn bar(attr: TokenStream, item: TokenStream) -> TokenStream {
    // ...
}

pub struct TokenStream(_);

impl TokenStream {
    pub fn empty() -> TokenStream;
    pub fn is_empty(&self) -> bool;
}

impl Clone for TokenStream { ... }
impl Debug for TokenStream { ... }
impl Display for TokenStream { ... }
impl FromStr for TokenStream { ... }
impl From<TokenTree> for TokenStream { ... }
impl FromIterator<TokenTree> for TokenStream { ... }
impl FromIterator<TokenStream> for TokenStream { ... }
impl !Send for TokenStream { ... }
impl !Sync for TokenStream { ... }

impl IntoIterator for TokenStream {
    type Item = TokenTree;
    type Iter = token_stream::IntoIter;
}

pub mod token_stream {
    pub struct IntoIter(_);

    impl Iterator for IntoIter {
        type Item = ::TokenTree;
    }
}

pub enum TokenTree {
    Op(Op),
    Term(Term),
    Literal(Literal),
    Group(Group),
}

impl TokenTree {
    pub fn span(&self) -> Span;
    pub fn set_span(&mut self, span: Span);
}

impl Clone for TokenTree { ... }
impl Debug for TokenTree { ... }
impl Display for TokenTree { ... }
impl From<Op> for TokenTree { ... }
impl From<Term> for TokenTree { ... }
impl From<Literal> for TokenTree { ... }
impl From<Group> for TokenTree { ... }
impl !Send for TokenTree { ... }
impl !Sync for TokenTree { ... }

pub struct Span(_);

impl Span {
    pub fn call_site() -> Span;
}

impl Clone for Span { ... }
impl Copy for Span { ... }
impl Debug for Span { ... }
impl !Send for Span { ... }
impl !Sync for Span { ... }

pub struct Group(_);

pub enum Delimiter {
    Parenthesis,
    Brace,
    Bracket,
    None,
}

impl Group {
    pub fn new(delimiter: Delimiter, stream: TokenStream) -> Group;
    pub fn stream(&self) -> TokenStream;
    pub fn delimiter(&self) -> Delimiter;
    pub fn span(&self) -> Span;
    pub fn set_span(&mut self, span: Span);
}

impl Clone for Group { ... }
impl Debug for Group { ... }
impl Display for Group { ... }
impl !Send for Group { ... }
impl !Sync for Group { ... }

impl Copy for Delimiter { ... }
impl Clone for Delimiter { ... }
impl Debug for Delimiter { ... }
impl PartialEq for Delimiter { ... }
impl Eq for Delimeter { ... }

pub struct Term(_);

impl Term {
    pub fn new(s: &str, span: Span) -> Term;
    pub fn span(&self) -> Span;
    pub fn set_span(&mut self, span: Span);
}

impl Copy for Term { ... }
impl Clone for Term { ... }
impl Debug for Term { ... }
impl Display for Term { ... }
impl !Send for Term { ... }
impl !Sync for Term { ... }

pub struct Op(_);

pub enum Spacing {
   Alone,
   Joint,
}

impl Op {
    pub fn new(op: char, spacing: Spacing) -> Op;
    pub fn op(&self) -> char;
    pub fn spacing(&self) -> Spacing;
    pub fn span(&self) -> Span;
    pub fn set_span(&mut self, span: Span);
}

impl Debug for Op { ... }
impl Display for Op { ... }
impl Clone for Op { ... }
impl Copy for Op { ... }
impl !Send for Op { ... }
impl !Sync for Op { ... }

impl Copy for Spacing { ... }
impl Clone for Spacing { ... }
impl Debug for Spacing { ... }
impl PartialEq for Spacing { ... }
impl Eq for Spacing { ... }

pub struct Literal(_);

impl Literal {
  // panic on infinity and NaN
  pub fn f{32,64}_{un,}suffixed(f: f{32,64}) -> Literal;

  pub fn i{8,16,32,64,128,size}_{un,}suffixed(n: i{8,16,32,64,128,size}) -> Literal;
  pub fn u{8,16,32,64,128,size}_{un,}suffixed(n: u{8,16,32,64,128,size}) -> Literal;

  pub fn string(s: &str) -> Literal;
  pub fn character(c: char) -> Literal;
  pub fn byte_string(b: &[u8]) -> Literal;

  pub fn span(&self) -> Span;
  pub fn set_span(&mut self, span: Span) -> Span;
}

impl Clone for Literal { ... }
impl Debug for Literal { ... }
impl Display for Literal { ... }
impl !Send for Literal { ... }
impl !Sync for Literal { ... }

Team member @alexcrichton has proposed to merge this. The next step is review by the rest of the tagged teams:

• [x] @Kimundi
• [ ] @SimonSapin
• [x] @alexcrichton
• [x] @aturon
• [x] @cramertj
• [x] @dtolnay
• [x] @eddyb
• [x] @joshtriplett
• [x] @nikomatsakis
• [x] @nrc
• [ ] @pnkfelix
• [x] @scottmcm
• [x] @sfackler
• [x] @withoutboats

No concerns currently listed.

Once a majority of reviewers approve (and none object), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

cc @rust-lang/compiler, y'all I know are also very interested in this and please feel free to raise any objections. If you have a blocking objection I can also register it for you

It looks like, using the provided API proposed for stabilization, proc macros can easily generate all sorts of tokens, or copy them from input to output, but cannot easily inspect a stream of tokens at even a superficial level. For instance, Literal does not implement Eq or PartialEq. Any particular reason for that? (And even that would only allow comparison with specified constant values, rather than extracting a value and processing it in code.)

Inside a proc macro implementation, given a TokenStream, what can a proc macro do to introspect a literal? Or, for that matter, extract the value of one? Is this simply not supported yet? Is there a plan to support it in the future?

(I'm not attempting to block stabilization of the proposed subset; I'd just like to better understand that subset and its intended features.)

@alexcrichton New items will have the same hygiene as macro_rules! does today. They will
not be hygienic. New items added to the AST will enter the same namespace as
other items in the module.

I'm not on the Rust core team, and I haven't thoroughly looked at all of the past discussion, but this sounds really bad to me. What if the macro wants to generate helper items which can only be accessed by the macro?

Rather than throwing out hygiene, I think it's better to enforce full 100% hygiene, but provide a way for macros to explicitly opt-out of hygiene for a particular variable.

@joshtriplett you would use the Display impl for Literal.

@joshtriplett yeah as @dtolnay mentioned you'd use Display which is what a crate like literalext is built on. In general the API is intended to be the "bare minimum", not a first-class API which has all the bells and whistles. Those will likely come later!

This round of stabilization is intended to be "maximally minimal" in the sense that it provides the least amount of functionality to cover all the use cases (aka Display for Literal and nothing else to interpret it).

@Pauan it's not great, yeah, but it's the exact same as macro_rules! which has been stable for years now and working in many different contexts. We want to enable hygiene and better treatment here but we're not quite ready yet. The Span::call_site() constructor is the lynchpin here where by using that you're requesting "no hygiene". Eventually we'll stabilize more options.

Unfortunately "full 100% hygiene" is years away, and the purpose of this proposal is to get something stable for Rust 2018

@alexcrichton So is the plan to stabilize it now with a hygiene hole and then a couple epochs/eras/editions later fix that hygiene hole?

Since Span::call_site() exists, is there some reason that it can't have full hygiene right now? Is it a lack of manpower/time, or are there theoretical/semantic concerns that still need to be fleshed out in an RFC?

@Pauan to reiterate there's no real "hole" in the sense that it already exists today. This is the exact same hygiene story with macro_rules! and custom derive. This proposal is effectively an extension of that, not introducing anything new.

Given a "perfect hygiene story" we'll always want some form of opt-out. Currently that opt-out is Span::call_site() (effectively). By only stabilizing that we're only stabilizing an opt-out mechanism for hygiene. Eventually we'll stabilize more functionality to make items fully hygienic where necessary.

Stabilizing hygiene is much further away. There are (AFAIK) unresolved research questions, it's not a question of manpower.

@dtolnay Ah, I see. So, you generate text and then re-parse the text? I suppose that works for the time being.

cc @rust-lang/compiler, y'all I know are also very interested in this and please feel free to raise any objections. If you have a blocking objection I can also register it for you

I still have a few things to review and a PR with couple proc-macro API tweaks in progress.

This is the exact same hygiene story with macro_rules! and custom derive.

Sure, but I thought one of the goals was to make a better macro system, so pointing to the existing (broken and limited) system doesn't sound very convincing to me.

Given a "perfect hygiene story" we'll always want some form of opt-out.

Absolutely, but my point is that since it's possible to use Span::call_site() to break hygiene (and thus gain the macro_rules! functionality) if desired, by making it hygienic by default you gain access to both hygiene and non-hygiene. Whereas if it stabilizes with non-hygiene then it's stuck with only non-hygiene.

Eventually we'll stabilize more functionality to make items fully hygienic where necessary.

That sounds like a rather annoying (and confusing) band-aid: proc macros would be hygienic in most situations (but not items), but you can then use a new feature to opt-in to hygiene for items. Am I understanding that right?

There are (AFAIK) unresolved research questions, it's not a question of manpower.

Okay, that's fair enough. I understand the desire for expediency and pragmaticism. As long as the plan is to eventually have solid hygiene by default in a later epoch/era/edition, then I'm okay with temporarily having bad hygiene. I just don't want us to stabilize something that we'll regret later.

In a sense, it already is "hygienic by default," it's just that the default isn't supported yet. Span::call_site is the opt-out. :)

@Pauan To elaborate on what @alexcrichton said:

Constructing an identifier requires a Span, which contains hygiene information. (See Term::new)

In the future, we may expose many different ways to construct Spans, which reflect different options for hygiene. But for now, we will only expose Span::call_site which takes the scope of the caller.

This way, we can avoid stabilizing the whole hygiene system right away, while keeping open the ability to add hygiene later. I think it's a pretty clever trick to get around the issue!

@rpjohnst So you mean that the only way to return new items from a proc macro is to use Span::call_site?

If so, then that sounds perfect: in that case hygiene can indeed be added in the future in a clean and backwards compatible way.

@lfairy Ahhh, I see, I hadn't realized that it was necessary to use Span::call_site, I thought there was an implicit default unhygienic scope. Thank you for the explanation! You are right that is a very clever trick.

As such, I have no more objections.

@alexcrichton

Attributes, macro_rules, and procedural
macros all reside in the maro namespace.

Last I checked with #[feature(proc_macro)], it had a backward-incompatible change in stable code for the derive-error-chain crate as described here. From the quoted statement, it seems like it'll continue to be like this in in macros 1.2. Is that correct?

@Arnavion oh dear, that sounds bad! We can't really do any big backward-incompatible changes, so we'll have to figure out a way to get that to keep working. One possible solution would be to defer "this isn't an attribute" errors until after #[derive] expansion.

It appears that #48644 and #47786 are done. Could someone update the OP?

@alexcrichton Has there been any discussion on allowing access to the internal data of Literal and friends? I get that they're not enums anymore for backwards compatibility reasons but is there any reason we can't have something like this for Literal to start with:

    impl Literal {
        pub fn as_str(&self) -> Option<&str> {}
        pub fn to_int(&self) -> Option<i64> {}
        pub fn to_uint(&self) -> Option<u64> {}
        pub fn to_float(&self) -> Option<f64> {}
    }

We could even have a bunch of TryFrom impls for the different literal subtypes.

@abonander it's been discussed yeah but the conclusion has been to add them later rather than make them necessary for the first pass of stabilization. They can be built on crates.io with the Display implementation and we've got room to add the later in the future

:bell: This is now entering its final comment period, as per the review above. :bell:

@alexcrichton
!Send and !Sync impls for proc macro API types are not written explicitly, but inferred.
For some structs (e.g. Op) the inferred impl differ from what is specified in https://github.com/rust-lang/rust/issues/38356#issuecomment-383693017.
It probably makes sense to add the impls explicitly.

Oops yes indeed! I've opened https://github.com/rust-lang/rust/pull/50453 to address that

@alexcrichton We have a nasty issue with Term::as_str: I was trying to sketch an informal proof that due to strict scoping, &'a Term -> &'a str can be implemented to borrow into some scoped interner, and 'a could never be larger than the scope of the interner itself (which only matters if the function succeeds, i.e. if it's called within that scope that the interner is live during).

AFAICT, Term::as_str is sound, but only assuming that 'a condition.

Which is upheld by e.g. thread_local!, because, while it lets you escape the Term value, it gives you very short-lived 'as, which if Term::as_str succeeds, are strictly shorter than the interner scope.
In general, the way proc_macro expansion happens, and because Term is thread-local, there are very few ways of escaping Term, and I assumed thread_local! was effectively the only one.

But Box::leak also exists! It's still unstable, but today, Box::leak(Box::new(term)).as_str() returns &'static str. I wonder if there are other abstractions broken by Box::leak (cc @RalfJung)

OTOH, this is only complicated because Term can't own the string data, as it's Copy.
If we remove the Copy on Term, we could keep a lazy Option<Cell<Rc<String>> in there.

@eddyb oh Term::as_str is slated for removal and wasn't included in the stabilization here, so nothing to worry about! It's just sticking around so we don't break proc-macro2, but once this stabilizes we can release a breaking change for proc-macro2

@eddyb I don't know the context here, but Box::leak is justified in my formal model. If you own some memory forever (i.e., in a box), you can totally say that it has any lifetime.

I have a few questions:

1. Is quote! not being stabilized?
2. Will quote! have the proc_macro_non_items feature gate applied to it? Commit 79630d4fdfc775b241cae0a209edec2687a29f0f now requires it, but quote! is still marked #[unstable(feature = "proc_macro" ....
3. Will tracking issues be filed for the stabilization of proc_macro_path_invoc, proc_macro_mod, proc_macro_expr, and proc_macro_non_items?

_{Unrelated side question: where is quote! implemented? It's clearly not implemented here, and since rustc is telling me it's a procedural macro (see proc_macro_non_items), it rules out here.}

@mjbshaw Sooo fun story: it is implemented in proc_macro::quote.
rustc_metadata pretends that any crate named proc_macro contains a procedural macro named quote in it, but the implementation used is from the proc_macro rustc_metadata links against.

I still have a few things to review and a PR with couple proc-macro API tweaks in progress.

Review PR: https://github.com/rust-lang/rust/pull/50473

Feature request for the 1.2 API: add a delimiter for angle brackets (</>) so things like <T> (in fn foo<T>() {}) will be parsed as a Group. Not doing this makes parsing e.g. generics unnecessarily complicated.

@mjbshaw I don't think it would work at the token level to try to determine whether two < > are grouped with each other. For example in the following macro input:

m!($A<$B, $C>=$D);

Maybe this is two boolean expressions $A < $B and $C >= $D, or maybe it represents generics in a type alias for example expanding to type $A <$B,$C> = $D;.

assert_both!(a<AMAX, b>=BMIN);

define_type_alias!(SwappedResult<E, T>=std::result::Result<T, E>);

I've updated the OP slightly but we've got what look to me like two possible showstopper bugs related to hygiene, depending on how they turn out:

• Hygiene is weird
• Hygiene is weirder

For those following this thread the API is likely to be changed from the original FCP comment in https://github.com/rust-lang/rust/pull/50473 proposed by @petrochenkov. The summary so far is:

• Rename Term::new to Term::ident and validate the input
• Add Term::lifetime and validate the input
• Add Term::raw_ident and validate input
• Rename Op to Punct
• Validate input on Punct::new
• RenameOp::op to Punct::as_char

One minor API request I would make (possibly in #50473 -- @petrochenkov) is some way to append tokens onto a TokenStream. Possibly:

impl Extend<TokenTree> for TokenStream

I believe this would make it possible to eliminate the quote::Tokens type (which is basically Vec<TokenTree>) and reduce the proliferation of different types in the ecosystem that all mean "some tokens" -- tracked in https://github.com/dtolnay/syn/issues/205.

+1 on @dtolnay’s above suggestion.

The final comment period, with a disposition to merge, as per the review above, is now complete.

I have two questions:

1. As already asked here, what's about quote!? What should be the default way to create the final TokenStream? Should it be done manually? Or should one use the quote crate? Is proc_macro::quote! supposed to be stabilized at some point in the future?

2. Is my understanding correct in that the difference between declaring a function-like macro and an attribute-like macro is just the number of arguments? I.e.:

   /// Invoked as `foo!()`
   #[proc_macro]
   pub fn foo(a: TokenStream) -> TokenStream {
       // ...
   }
   
   /// Invoked as `#[bar]`
   #[proc_macro]
   pub fn bar(attr: TokenStream, item: TokenStream) -> TokenStream {
       // ...
   }
   

   The only difference is that one takes one TokenStream as argument, where the other one takes two. Isn't this a bit subtle? Why not use #[proc_macro_attribute] and #[proc_macro_function_like] instead? Has this been discussed somewhere? If so, I'd be happy if someone could link the discussion.

Thanks for your work on this! :)

@LukasKalbertodt Attributes are currently declared with #[proc_macro_attribute]. I don't know if there is a deliberate intention to change that or if it is a typo in the FCP proposal.

Issue: Tokens parsed from strings do not get Span::call_site spans: https://github.com/rust-lang/rust/issues/50050#issuecomment-390520317.

I think we need to change this and also change what Span::call_site returns to fix macro backtraces, clippy and edition hygiene for tokens using call-site spans.

@LukasKalbertodt the quote! macro in the proc_macro crate is not being stabilized as part of this FCP, but the quote crate on crates.io builds on the API proposed here. Also as @abonander pointed out attribute macros are declared with #[proc_macro_attribute] and #[proc_macro] is reserved exclusively for foo!()-style macros

Shouldn't #[proc_macro_attribute] be named #[proc_attribute_macro] or #[attribute_proc_macro]?

@Zoxc We already have #[proc_macro_derive] stable, so it'd be strange to not follow that for attribute macros.

Can we get PartialEq<char> and PartialEq<Punct> for Punct (similar to Ident's PartialEq implementations)? It seems like it should be pretty safe to add. I'm happy to write the PR, but don't want to if the idea is a no-go.

@mjbshaw Would PartialEq<Punct> also compare spans? PartialEq<char> seems fine, OTOH.

@eddyb PartialEq for Ident doesn't compare spans (I know this is proc-macro2 source and not proc-macro source). I'm ambivalent as to whether spans are included in the comparison, but I would expect Punct and Ident to behave similarly in this respect. Punct also has its own Spacing, which I presume would be included in the comparison (though maybe others think differently).

I'd be fine only implementing PartialEq<char> for Punct for now. PartialEq<Punct> can be hashed out later.

@mjbshaw the proc-macro2 crate has the freedom of not being exactly what proc_macro is upstream and gives us some freedom to experiment with various tweaks. I suspect that if it works out well on crates.io we can add it to proc_macro, but it's of course backwards-compatible to add to proc_macro so we're starting with the bare minimum and we can scale up from there once it's stable.

I've done some triage work for all bugs related to macros 1.2. Most bugs can be classified into "serious bugs" or "all span related bugs". Currently span-related bugs purely affect error messages (as the intention of macros 1.2 is not to change resolution). The remaining non-span-related bug (aka serious) is https://github.com/rust-lang/rust/issues/50050, which @petrochenkov has a solution for.

Gnome-class broke with the changes to proc_macro2/syn/quote, plus the feature gate for generating modules from proc macros. It's fixed now, fortunately.

What things should I monitor to keep on top of changes in this little ecosystem?

@federicomenaquintero If you use unstable features, consider having a regular CI job (daily, weekly, whatever works for you) that compiles your code with the latest Nightly and notify you if it fails. (Travis-CI supports enabling such a "cron job" in its settings.)

@SimonSapin thanks, that's a good idea. We pin versions of those crates in our Cargo.toml, but it may be time to remove the version numbers and just let Cargo download the latest. Is this the correct way to do it?

@federicomenaquintero This is increasingly off-topic so if you’d like to continue this discussion please do so somewhere else such as on IRC or http://users.rust-lang.org/, but the general recommendation is that applications (as opposed to libraries) should ship Cargo.lock together with their source code, which effectively pins dependencies. In Cargo.toml, declaring a dependency like foo = "1.2.3" is recommended, it implicitly means "that version or later, if compatible according to SemVer".

So, I've just run into an issue with procedural macros that is blocking me in developing a crate I want to make.

Consider the following:

#[my_attribute]
struct MyStruct {
  #[other_attribute]
  field: String,
}

This currently doesn't work because both the features proc_macro and custom_attributes are required but can't be used at the same time. As I understand stabilisation of proc macros would eliminate the need for the feature flags?

The other thing is that this way #[my_attribute] could potentially generate code that prevents #[other_attribute] from ever being run. What would be really cool if on the "outer" attribute, I could register inner attributes that work the same way that #[derive(Foo)] work.

What's about @SimonSapin's comment about the dummy session?

Would it be possible/desirable to replace this panic with implicitly creating a dummy "session"? Or perhaps add a public API (with a path to stabilization) to create one?

I think it would be very useful to have a dummy session. Writing unit test for proc-macro crates is pretty much impossible or at least very inconvenient otherwise. Furthermore, it's not only TokenStream::from_str, but also other functions requiring a session.

@alexcrichton

I've done some triage work for all bugs related to macros 1.2. Most bugs can be classified into "serious bugs" or "all span related bugs".

I'd like to upgrade https://github.com/rust-lang/rust/issues/50504 to the "serious" status - the module issue described there is only a symptom of a deeper problem - expansion IDs for proc macros not being registered properly. I don't know what other consequences this may have.
There's a PR fixing the underlying issue (https://github.com/rust-lang/rust/pull/51952), but there are regressions from the fix and I haven't looked through them yet.

I've posted a PR to stabilize more of procedural macros at https://github.com/rust-lang/rust/pull/52081

@petrochenkov sounds good to me, I'll comment on the PR

I just posted on the macro naming tracking issue about issues with proc_macro_derive child attributes and how they interact with the naming system. There is another issue with the way proc_macro_derive child attributes interact with scoping and naming, but seems more relevant to bring up here. Since paths in attributes are not currently on the course to stabilization, we get the possibility that #[derive(foo::Parent)] could have a child attribute #[foo::Child], but the derivation macro would have no way, on its face, to identify if the child attribute is actually its own child, as it cannot perform name lookup. For now, I don't have an easy solution, but I think that it is something that should be on the radar for the future of interdependent attributes; there is no reason that proc_macro_attribute attributes couldn't want to interact in a way that runs into similar lookup issues.

I tried to compile my project today, but something is broken which is probably related to this issue. All the error messages had the message: "(see issue #38356)". This is how I got here.
I include here the error message I got during compilation. I also include my Cargo.toml.

I find this very surprising, as my project is pinned to a specific Rust nightly version (rustc 1.29.0-nightly (9bd8458c9 2018-07-09)) What could have changed? Possibly a library updated?

[[bin]]
name = "main"
path = "src/bin/main.rs"

[dependencies]
log = "0.4"
pretty_env_logger = "0.2"

rand = "0.4"
ring = "=0.13.0-alpha"
untrusted = "0.6"

bytes = "0.4"
futures = "0.1"
tokio-io = "0.1"
tokio-core = "0.1"
futures-await = "0.1"

capnp = "0.8"
rusqlite = "0.13"

async_mutex = { git = "https://github.com/realcr/async_mutex", rev = "a1d973ed7" }

num-bigint = "0.2.0"
num-traits = "0.2.4"

[dev-dependencies]

[dependencies.byteorder]
version = "1.1"
features = ["i128"]

[build-dependencies]
capnpc = "0.8"

[profile.release]
debug = true

$ cargo test
    Updating git repository `https://github.com/realcr/async_mutex`
   Compiling proc-macro2 v0.4.8                                                                                                                                                                                    
   Compiling cswitch v0.1.0 (file:///home/real/projects/d/cswitch)                                                                                                                                                 
error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)8: proc-macro2                                                                                                                        
  --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:33:40
   |
33 |     let works = panic::catch_unwind(|| proc_macro::Span::call_site()).is_ok();
   |                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:213:13
    |
213 |     Nightly(proc_macro::token_stream::IntoIter),
    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:438:11
    |
438 | impl From<proc_macro::Span> for ::Span {
    |           ^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:284:13
    |
284 |     Nightly(proc_macro::SourceFile, FileName),
    |             ^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:332:13
    |
332 |     Nightly(proc_macro::Span),
    |             ^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:461:13
    |
461 |     Nightly(proc_macro::Ident),
    |             ^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:523:13
    |
523 |     Nightly(proc_macro::Literal),
    |             ^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:116:47
    |
116 |                     Delimiter::Parenthesis => proc_macro::Delimiter::Parenthesis,
    |                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:117:43
    |
117 |                     Delimiter::Bracket => proc_macro::Delimiter::Bracket,
    |                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:118:41
    |
118 |                     Delimiter::Brace => proc_macro::Delimiter::Brace,
    |                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:119:40
    |
119 |                     Delimiter::None => proc_macro::Delimiter::None,
    |                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:122:33
    |
122 |                 let mut group = proc_macro::Group::new(delim, tt.stream.inner.unwrap_nightly());
    |                                 ^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:128:39
    |
128 |                     Spacing::Joint => proc_macro::Spacing::Joint,
    |                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:129:39
    |
129 |                     Spacing::Alone => proc_macro::Spacing::Alone,
    |                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:131:30
    |
131 |                 let mut op = proc_macro::Punct::new(tt.as_char(), spacing);
    |                              ^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:113:17
    |
113 |         let tt: proc_macro::TokenTree = match token {
    |                 ^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:238:13
    |
238 |             proc_macro::TokenTree::Group(tt) => {
    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:240:21
    |
240 |                     proc_macro::Delimiter::Parenthesis => Delimiter::Parenthesis,
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:241:21
    |
241 |                     proc_macro::Delimiter::Bracket => Delimiter::Bracket,
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:242:21
    |
242 |                     proc_macro::Delimiter::Brace => Delimiter::Brace,
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:243:21
    |
243 |                     proc_macro::Delimiter::None => Delimiter::None,
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:250:13
    |
250 |             proc_macro::TokenTree::Punct(tt) => {
    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:252:21
    |
252 |                     proc_macro::Spacing::Joint => Spacing::Joint,
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:253:21
    |
253 |                     proc_macro::Spacing::Alone => Spacing::Alone,
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:259:13
    |
259 |             proc_macro::TokenTree::Ident(s) => ::Ident::_new(Ident::Nightly(s)).into(),
    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:260:13
    |
260 |             proc_macro::TokenTree::Literal(l) => ::Literal::_new(Literal::Nightly(l)).into(),
    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:289:20
    |
289 |     fn nightly(sf: proc_macro::SourceFile) -> Self {
    |                    ^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:339:27
    |
339 |             Span::Nightly(proc_macro::Span::call_site())
    |                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:347:27
    |
347 |             Span::Nightly(proc_macro::Span::def_site())
    |                           ^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:369:30
    |
369 |     pub fn unstable(self) -> proc_macro::Span {
    |                              ^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:430:32
    |
430 |     fn unwrap_nightly(self) -> proc_macro::Span {
    |                                ^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:439:24
    |
439 |     fn from(proc_span: proc_macro::Span) -> ::Span {
    |                        ^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:468:48
    |
468 |             Span::Nightly(s) => Ident::Nightly(proc_macro::Ident::new(string, s)),
    |                                                ^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:475:48
    |
475 |             Span::Nightly(s) => Ident::Nightly(proc_macro::Ident::new_raw(string, s)),
    |                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:495:32
    |
495 |     fn unwrap_nightly(self) -> proc_macro::Ident {
    |                                ^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:583:30
    |
583 |             Literal::Nightly(proc_macro::Literal::f32_unsuffixed(f))
    |                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:591:30
    |
591 |             Literal::Nightly(proc_macro::Literal::f64_unsuffixed(f))
    |                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:599:30
    |
599 |             Literal::Nightly(proc_macro::Literal::string(t))
    |                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:607:30
    |
607 |             Literal::Nightly(proc_macro::Literal::character(t))
    |                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:615:30
    |
615 |             Literal::Nightly(proc_macro::Literal::byte_string(bytes))
    |                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:636:32
    |
636 |     fn unwrap_nightly(self) -> proc_macro::Literal {
    |                                ^^^^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/lib.rs:322:30
    |
322 |     pub fn unstable(self) -> proc_macro::Span {
    |                              ^^^^^^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:531:34
    |
531 |                   Literal::Nightly(proc_macro::Literal::$name(n))
    |                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
...
552 | /     suffixed_numbers! {
553 | |         u8_suffixed => u8,
554 | |         u16_suffixed => u16,
555 | |         u32_suffixed => u32,
...   |
565 | |         f64_suffixed => f64,
566 | |     }
    | |_____- in this macro invocation
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:543:34
    |
543 |                   Literal::Nightly(proc_macro::Literal::$name(n))
    |                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
...
568 | /     unsuffixed_integers! {
569 | |         u8_unsuffixed => u8,
570 | |         u16_unsuffixed => u16,
571 | |         u32_unsuffixed => u32,
...   |
578 | |         isize_unsuffixed => isize,
579 | |     }
    | |_____- in this macro invocation
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
  --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:45:34
   |
45 |             TokenStream::Nightly(proc_macro::TokenStream::new())
   |                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
  --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:53:46
   |
53 |             TokenStream::Nightly(tts) => tts.is_empty(),
   |                                              ^^^^^^^^
   |
   = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:123:23
    |
123 |                 group.set_span(span.inner.unwrap_nightly());
    |                       ^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:132:20
    |
132 |                 op.set_span(tt.span().inner.unwrap_nightly());
    |                    ^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:239:38
    |
239 |                 let delim = match tt.delimiter() {
    |                                      ^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:245:74
    |
245 |                 let stream = ::TokenStream::_new(TokenStream::Nightly(tt.stream()));
    |                                                                          ^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:247:58
    |
247 |                 g.set_span(::Span::_new(Span::Nightly(tt.span())));
    |                                                          ^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:251:40
    |
251 |                 let spacing = match tt.spacing() {
    |                                        ^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:255:43
    |
255 |                 let mut o = Punct::new(tt.as_char(), spacing);
    |                                           ^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:256:58
    |
256 |                 o.set_span(::Span::_new(Span::Nightly(tt.span())));
    |                                                          ^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:290:45
    |
290 |         let filename = stable::file_name(sf.path().to_string());
    |                                             ^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:304:44
    |
304 |             SourceFile::Nightly(a, _) => a.is_real(),
    |                                            ^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:355:69
    |
355 |             (Span::Nightly(a), Span::Nightly(b)) => Span::Nightly(a.resolved_at(b)),
    |                                                                     ^^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:363:69
    |
363 |             (Span::Nightly(a), Span::Nightly(b)) => Span::Nightly(a.located_at(b)),
    |                                                                     ^^^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:424:55
    |
424 |             (Span::Nightly(a), Span::Nightly(b)) => a.eq(b),
    |                                                       ^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:482:50
    |
482 |             Ident::Nightly(t) => Span::Nightly(t.span()),
    |                                                  ^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:489:56
    |
489 |             (Ident::Nightly(t), Span::Nightly(s)) => t.set_span(s),
    |                                                        ^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:623:56
    |
623 |             Literal::Nightly(lit) => Span::Nightly(lit.span()),
    |                                                        ^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error[E0658]: use of unstable library feature 'proc_macro' (see issue #38356)
   --> /home/real/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-0.4.8/src/unstable.rs:630:62
    |
630 |             (Literal::Nightly(lit), Span::Nightly(s)) => lit.set_span(s),
    |                                                              ^^^^^^^^
    |
    = help: add #![feature(proc_macro)] to the crate attributes to enable

error: aborting due to 63 previous errors

For more information about this error, try `rustc --explain E0658`.
error: Could not compile `proc-macro2`. 

Do you have an idea of what could have gone wrong, and how can it be fixed? Thanks!

@realcr for the proc-macro2 crate you can simply run cargo update and that should do the trick!

@alexcrichton I don't think that's the problem here. I rather think @realcr already updated proc-macro2 since the error message says proc-macro2-0.4.8 everywhere. The problem is rather that the nightly version is fixed to one that doesn't include #52081. I had the same problem today and wondered about why proc-macro2 only bumped the minor version. But I'm not very familiar with how proc-macro2 handles compatibility.

@realcr Try updating your nightly compiler or enforce a proc-macro-2 version < 0.4.8 in your project.

@alexcrichton , @LukasKalbertodt : Thanks for the quick reply.
I just updated my nightly compiler to the latest version. It did eliminate the proc-macro-2 problems, but I got lots of new compilation errors. Example:

error[E0277]: the trait bound `impl futures::Future: std::future::Future` is not satisfied
   --> src/networker/messenger/handler/handle_neighbor.rs:191:25
    |
191 |         let signature = await!(self.security_module_client.request_signature(failure_signature_buffer))
    |                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `std::future::Future` is not implemented for `impl futures::Future`
    |
    = note: required by `std::future::poll_in_task_cx`
    = note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)

...

error[E0627]: yield statement outside of generator literal
   --> src/networker/messenger/handler/handle_neighbor.rs:403:13
    |
403 | /             await!(self.reply_with_failure(remote_public_key.clone(), 
404 | |                                            channel_index,
405 | |                                            request_send_msg.clone()))?
    | |_____________________________________________________________________^
    |
    = note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)

To confirm, my current version of rustc:

rustc 1.29.0-nightly (1ecf6929d 2018-07-16)

In order to trace the source of the problem I tried to compile the basic example from futures_await but it stopped working too. I will file an issue there so that we can solve this problem.

@LukasKalbertodt https://github.com/alexcrichton/proc-macro2#unstable-features

Because you are using the unstable feature.

@realcr your new compilation problem is not related to this issue, please stay on topic.

@TeXitoi : Feel free to edit or remove anything if you think it is not relevant. I try my best to help you out, it is difficult for me to know what is on topic and what not. The error message "(see issue #38356)" is what brought me here.

I tried to upgrade my compiler version and I've got this error. My code

#![no_std]
#![feature(proc_macro)]
#![feature(proc_macro_gen)]
#![feature(custom_attribute)]
#![feature(alloc)]

#[macro_use(eth_abi)]
extern crate pwasm_abi_derive;

and error that says that I didn't use #![feature(proc_macro)], but I did!

error[E0658]: attribute procedural macros are experimental (see issue #38356)
  --> src\lib.rs:67:5
   |
8  | #[macro_use(eth_abi)]
   |             ------- procedural macro imported here
...
67 |     #[eth_abi(TokenEndpoint, TokenClient)]
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = help: add #![feature(proc_macro)] to the crate attributes to enable

@Pzixel you'll want to switch #![feature(proc_macro)] to #![feature(use_extern_macros)] and that should do the trick

I think you'll also need to use the module system to import procedural macros (and make sure you have an up-to-date nightly compiler)

@alexcrichton yes, I just figured it out, thanks to the article. However, it still not work:

error[E0433]: failed to resolve. Use of undeclared type or module `Vec`
  --> src\lib.rs:66:5
   |
66 |     #[eth_abi(TokenEndpoint, TokenClient)]
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Use of undeclared type or module `Vec`

error[E0412]: cannot find type `Vec` in this scope
  --> src\lib.rs:66:5
   |
66 |     #[eth_abi(TokenEndpoint, TokenClient)]
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ not found in this scope

Import rules for macros have changed too? Or I can't figure out why it started complain here.

@Pzixel that may be a bug in the procedural macro or the compiler, can you file a dedicated issue for that?

Well, I think I should rewrite my code first to make it at least looking working :) Right now it doesn't, you guys changed a lot with this feature. BRB when done. If it doesn't disappear I create an issue.

@alexcrichton
Do you know what "preprocessing" is applied to tokens before they are passed to a procedural macro?
I know that derives has at least $crate eliminated and cfg expanded before they get input tokens (plus roundtrips through strings are not entirely lossless, but that's fixable).

We should make sure none of this happens with newly stabilized procedural macros and they get input tokens precisely (modulo bugs).

@alexcrichton sorry, cargo expand doesn't work on this crate for some reason. Cannot confirm if problem is on my side or on compiler one. So I'l continue blaming myself until this possibility is not completely excluded.

@petrochenkov the expansion of proc-macros has been pretty well vetted to date so I'm much less worried about that than the name resolution pieces. I'm not personally aware of preprocessing, but I do know that there's an expansion order where derives run last, cfgs run first, and otherwise it's mostly iterative.

I agree though that it's good to audit for!

Maybe I missed something obvious. But, is there no way to call a function, use a type, etc. from the proc_macro-defining crate from a procedural macro expansion? (or any other crate known from the proc_macro crate, FWIW)

There are workarounds, but AFAIU they won't work if the crate is renamed by the user of the procedural macro.

@Ekleog proc-macro crates are typically only compiled for the purpose of building crates that use them. They are not runtime dependencies. You can think of the entire proc-macro crate as a sort of compiler plugin, not a "normal" library.

This is especially visible when cross-compiling: like build scripts, proc-macros are compile for the host platform, not the target one.

@SimonSapin I agree with you, but it may be really useful to delegate part of the work to a function provided by a crate, even if it is not the proc-macro crate (eg. in my link above, the X-derive crate attempts to use a function from the X crate). Because otherwise that would mean that all code generated by proc-macros have to be either self-contain or make assumptions on the state of the call site.

Then, I can understand that rustc is not ready yet for this kind of feature (because I guess it'd work best without the need to split proc-macro from non-proc-macro crates, which appears to be somewhere on the long-term roadmap). But I wonder, if the current interface using only TokenStreams is stabilized, will it be possible to retrofit this feature later? Wouldn't it need something like a PathToBeResolvedFromTopOfGeneratingProcMacroCrate token type? (which would be a breaking change if added later on, afaiu)

Maybe it’ll be possible to make things more flexible eventually, but that seems fairly far off.

In the meantime, if you’re a library author, consider having a foo-proc-macros or foo-derive crate for procedural macros, and a "normal" foo library that contains run-time code but also re-exports the procedural macros. That way the user-facing API can be kept to a single crate. This is what serde does (in some configs) https://github.com/serde-rs/serde/blob/v1.0.71/serde/src/lib.rs#L304

It's worth pointing out though that this workaround still doesn't address the issue of the user renaming the root crate (e.g. serde's tracking issue).

#![feature(proc_macro)] is marked as stabilized since 1.29.0. Is this intended?

@CAD97, definitely. See https://github.com/rust-lang/rust/commit/65f3007fa8a08daf77f2b8382a56eb80cb277131 and https://internals.rust-lang.org/t/help-stabilize-a-subset-of-macros-2-0/7252

@Ekleog, the TokenStream is a stream of TokenTrees and each TokenTree has associated Span, which carries the scoping information. Except there is currently no way to create a Span for any other scope than “call site” (or empty). Basically coming up with a reasonably ergonomic way of creating Spans referring to specific crate is what is needed.

The reason I asked is that the checkbox isn't ticked. Would be nice to tick it off then!

With #![feature(proc_macro)] stabilized, what's left of this issue?

@jan-hudec Oh, I thought Spans were only for error reporting, as early blog posts mentioned a Hygiene (or similarly named) struct that played this role. I had assumed that these had vanished, and was apparently wrong. Thanks! :)

With #![feature(proc_macro)] stabilized, what's left of this issue?

Ideally, fresh issues need be filed for all individual remaining issues and not stabilized features, and then this issue can be closed (in the same way like it was done for https://github.com/rust-lang/rust/issues/44660).

Oh, I thought Spans were only for error reporting, as early blog posts mentioned a Hygiene (or similarly named) struct that played this role. I had assumed that these had vanished, and was apparently wrong. Thanks! :)

IIUC, Spans are the primary way of tracking hygiene context.

@mark-i-m Kind of. They include both source code location information (for user-facing messages/diagnostics) and syntax context (i.e. hygiene information).

Given the amount of discussion/traffic this issue gets, does it make sense to move proc_macro_diagnostic to its own tracking issue? I'd also like to figure out what the blockers are for the stabilization of that feature, and see if we can push it through. Diesel uses it, and it's been great so far. The lack of this feature on stable is leading to the community creating funky workarounds, like syn's newest version using compile_error! in its place.

@sgrif I've opened up such an issue: https://github.com/rust-lang/rust/issues/54140.

So I'm interesting in helping to stabilise the methods in Span and the LineColumn struct. I'll look at the open issues in the first comment, but if anyone wants to point a newbie to the compiler in a particular direction I'd appreciate it :+1:

The proc_macro_gen feature gate pointed me here, but in the check list at the top I see nothing that obviously refers to a (proc_)macro that generates other macro definitions. Has this been accounted for (other than in rustc's feature gate)?

@jjpe it's probably best at this time that we spin off dedicated tracking issues for those feature gates, this issue was largely dedicated to the initial wave of stabilizations

@alexcrichton I'm perfectly fine with that, it's more that in the process of looking at the proc_macro_gen feature, it referred me here only to find as good as no mention of it. That's a bit odd to me, so I decided to mention it, at least.

@xieyuheng What would CodeString/Code even look like i.e. what would its semantics be?
Keep in mind that TokenStream is nothing more than the source code verbatim except as a series of token values rather than bothersome text.

The extended API for TokenStream (with TokenTree) is stable in 1.29. Importing function-like proc macros will be stable in 1.30.

It seems that since rustc 1.30.0-nightly (63d51e89a 2018-09-28) it is no longer possible to traverse code inside module in separate file. If you process mod module;, you get TokenStream containing mod module;, WYSIWYG.

I understand, that this is necessary in order to preserver spans, hygiene and consistency with processed code. Is there or will be any way to work with content of modules in separate files? Its lack may be confusing for end users when trivial refactoring of module organization causes change in macro behavior.

Ok this issue is massive and has gotten to the point that I don't think it's too useful to keep open any more and track APIs. To that end I've opened https://github.com/rust-lang/rust/pull/54728 which splits this issue into a number of more fine-grained tracking issues:

• https://github.com/rust-lang/rust/issues/54722 - the quote! macro
• https://github.com/rust-lang/rust/issues/54723 - Ident::new_raw
• https://github.com/rust-lang/rust/issues/54724 - Span::def_site
• https://github.com/rust-lang/rust/issues/54140 - proc_macro and diagnostics
• https://github.com/rust-lang/rust/issues/54725 - inspecting a proc_macro::Span
• https://github.com/rust-lang/rust/issues/54726 - allowing custom inner attributes
• https://github.com/rust-lang/rust/issues/54727 - a sort of catch-all for "hygiene 2.0" reports, may want to be sharded itself.

At this point I'm going to close this, but if I've forgotten to split off any other tracking issues please let me know! I can certainly open up some more follow-ups

@alexcrichton What about of sub-attributes for attribute-like macro?
https://github.com/rust-lang/rust/issues/38356#issuecomment-397095541
Is there a issue for this?

@XX
Such sub-attributes don't necessarily have to be a language feature?
In case of derive registration of custom attributes is needed because derive macros cannot remove attributes from their input (the input is immutable).
Attribute macros can remove #[other_attribute]s from its input, so they will never get to name resolution and never report the "unresolved attribute" error.

(Beside that classic legacy unstable custom attributes mentioned in https://github.com/rust-lang/rust/issues/38356#issuecomment-397095541 are now compatible with proc macros.)

@petrochenkov Yes, thanks for the clarification.