A partial rewrite of splitarg and combinearg to fix https://github.com/FluxML/MacroTools.jl/issues/165 and https://github.com/FluxML/MacroTools.jl/issues/177.

The big change here is that arg_type can be nothing if no type is declared for an argument. I know that this is a change to the interface, but I also see no other way to fix the problem. While we are changing the interface, we may want to consider handling literal nothing as an argument default through the base Some and something interface. (discussed in https://github.com/FluxML/MacroTools.jl/issues/35).

There are a couple of problems with the current animal name scheme used in alias_gensyms:

1. There are only about 220 animal names in animals.txt. If you have more than 220 gensyms the following line in the current alias_gensyms function is going to error out:

[email protected]!(syms, x, pop!(left))

(left is a copy of the animals symbol list)

2. Currently, the animal names list is randomized when the package is initialized. While working on ONNX.jl we were finding that given the same inputs we would get different code generated due to this randomization. This makes it very difficult for us to debug as we made changes to ONNX.jl and wanted to compare code generated from a prior version on the same inputs.

This patch generates a simple unique id in the form of "sym_id_$counter" where the counter value is incremented on each gensym encountered.

I would have preferred replacing all of the "#" characters in the gensym with blanks, however that did not work for me even though I tried the same scheme in the REPL on a gensym string and it did work (see the comment I added in alias_gensyms - maybe someone could get that working?)

I also have a function that uses MacroTools to parse any function definition and return (name, arg, kwargs, body, return_type). Would that be accepted?

Previously combinedef would put an ::Any return type annotation on any function it reconstructed that didn't have one originally, and also wrap the function body in one extra begin block. These both seem completely superfluous except amazingly when combined they trigger the bug I reported here: https://github.com/JuliaLang/julia/issues/29326.

This PR gets rid of those things and more faithfully reconstructs the original function.

I also switched to returning the expression via @q which would allow whatever macro is calling combinedef to insert the desired line numbers in there.

This package is heavily used throughout the ecosystem so optimizing its load time is quite important. The compilation time of the shuffle! call in the __init__ function adds quite a bit of overhead, so I moved this to be done the first time the variable is actually used instead.

Without this PR:

julia> @time using MacroTools
  0.067275 seconds (142.56 k allocations: 9.640 MiB, 61.33% compilation time)

With this PR:

julia> @time using MacroTools
  0.024947 seconds (28.79 k allocations: 3.159 MiB, 22.04% compilation time)

It also removed quite a bit of "noise" when profiling other packages load time that happens to depend on this pacakge recursively.

I ran JET analysis on this package, and fixed some true positive errors.

run JET on MacroTools

julia> using JET
julia> report_file("src/MacroTools.jl"; analyze_from_definitions = true, annotate_types = true)

Before this PR

═════ 11 possible errors found ═════
┌ @ src/match/match.jl:27 Base.getproperty(Base.getproperty(MacroTools.Base, :match::Symbol)::typeof(match)(r"^@?(.*?)_+(_str)?$", MacroTools.string(s::Symbol)::String)::Union{Nothing, RegexMatch}, :captures::Symbol)
│┌ @ Base.jl:33 Base.getfield(x::Nothing, f::Symbol)
││ type Nothing has no field captures
│└──────────────
┌ @ src/match/types.jl:20 MacroTools.map(MacroTools.totype, ts::Vector{Symbol})
│┌ @ abstractarray.jl:2301 Base.collect_similar(A::Vector{Symbol}, Base.Generator(f::typeof(MacroTools.totype), A::Vector{Symbol})::Base.Generator{Vector{Symbol}, typeof(MacroTools.totype)})
││┌ @ array.jl:620 Base._collect(cont::Vector{Symbol}, itr::Base.Generator{Vector{Symbol}, typeof(MacroTools.totype)}, Base.IteratorEltype(itr::Base.Generator{Vector{Symbol}, typeof(MacroTools.totype)})::Base.EltypeUnknown, Base.IteratorSize(itr::Base.Generator{Vector{Symbol}, typeof(MacroTools.totype)})::Base.HasShape{1})
│││┌ @ array.jl:710 Base.collect_to_with_first!(Base._similar_for(c::Vector{Symbol}, Base.typeof(v1::Union{Expr, Symbol})::Union{Type{Expr}, Type{Symbol}}, itr::Base.Generator{Vector{Symbol}, typeof(MacroTools.totype)}, isz::Base.HasShape{1})::Union{Vector{Expr}, Vector{Symbol}}, v1::Union{Expr, Symbol}, itr::Base.Generator{Vector{Symbol}, typeof(MacroTools.totype)}, st::Int64)
││││┌ @ array.jl:715 Base.setindex!(dest::Vector{Symbol}, v1::Expr, i1::Int64)
│││││┌ @ array.jl:853 Base.convert(_::Type{Symbol}, x::Expr)
││││││ no matching method found for call signature: Base.convert(_::Type{Symbol}, x::Expr)
│││││└────────────────
││││┌ @ array.jl:715 Base.setindex!(dest::Vector{Expr}, v1::Symbol, i1::Int64)
│││││┌ @ array.jl:853 Base.convert(_::Type{Expr}, x::Symbol)
││││││ no matching method found for call signature: Base.convert(_::Type{Expr}, x::Symbol)
│││││└────────────────
┌ @ src/utils.jl:21 Base.collect(Base.Generator(#11::MacroTools.var"#11#12", MacroTools.map(MacroTools.esc, xs::Tuple)::Any)::Base.Generator{_A, MacroTools.var"#11#12"} where _A)
│┌ @ array.jl:697 Base.collect_to_with_first!(Base._array_for(Base.typeof(v1::Expr)::Type{Expr}, Base.getproperty(itr::Base.Generator{_A, MacroTools.var"#11#12"} where _A, :iter::Symbol)::Any, isz::Any)::Any, v1::Expr, itr::Base.Generator{_A, MacroTools.var"#11#12"} where _A, st::Any)
││┌ @ array.jl:721 Base.grow_to!(dest::Any, itr::Base.Generator{_A, MacroTools.var"#11#12"} where _A, st::Any)
│││┌ @ dict.jl:153 Base.indexed_iterate(Core.getfield(Base.indexed_iterate(y::Tuple{Expr, Any}, 1)::Tuple{Expr, Int64}, 1)::Expr, 1)
││││┌ @ tuple.jl:89 Base.iterate(I::Expr)
│││││ no matching method found for call signature: Base.iterate(I::Expr)
││││└───────────────
┌ @ src/utils.jl:423 MacroTools.rebuilddef(MacroTools.striplines(dict::Dict)::Dict)
│ variable MacroTools.rebuilddef is not defined: MacroTools.rebuilddef(MacroTools.striplines(dict::Dict)::Dict)
└────────────────────
┌ @ src/utils.jl:455 Core.tuple(splitvar::MacroTools.var"#splitvar#35"(arg::Any)::Union{Nothing, Tuple{Any, Any}}, Core.tuple(is_splat::Bool, default::Any)::Tuple{Bool, Any}...)
│ no matching method found for call signature: Core.tuple(splitvar::MacroTools.var"#splitvar#35"(arg::Any)::Union{Nothing, Tuple{Any, Any}}, Core.tuple(is_splat::Bool, default::Any)::Tuple{Bool, Any}...)
└────────────────────
┌ @ src/utils.jl:457 Core.tuple(splitvar::MacroTools.var"#splitvar#35"(arg_expr2::Any)::Union{Nothing, Tuple{Any, Any}}, Core.tuple(is_splat::Bool, MacroTools.nothing)::Tuple{Bool, Nothing}...)
│ no matching method found for call signature: Core.tuple(splitvar::MacroTools.var"#splitvar#35"(arg_expr2::Any)::Union{Nothing, Tuple{Any, Any}}, Core.tuple(is_splat::Bool, MacroTools.nothing)::Tuple{Bool, Nothing}...)
└────────────────────
┌ @ src/structdef.jl:13 MacroTools.parse_error(ex::Any)
│ variable MacroTools.parse_error is not defined: MacroTools.parse_error(ex::Any)
└───────────────────────
┌ @ src/structdef.jl:21 MacroTools.parse_error(ex::Any)
│ variable MacroTools.parse_error is not defined: MacroTools.parse_error(ex::Any)
└───────────────────────
┌ @ src/structdef.jl:29 MacroTools.parse_error(ex::Any)
│ variable MacroTools.parse_error is not defined: MacroTools.parse_error(ex::Any)
└───────────────────────
┌ @ src/examples/destruct.jl:24 MacroTools.error("Can't destructure fields with default values")
│┌ @ error.jl:33 error(::String)
││ may throw: Base.throw($(Expr(:invoke, MethodInstance for ErrorException(::String), :(Base.ErrorException), Core.Argument(2)))::ErrorException)
│└───────────────

After this PR: only the false positives ramain (Julia's inference or JET itself should be improved)

═════ 4 possible errors found ═════
┌ @ src/match/types.jl:20 MacroTools.map(MacroTools.totype, ts::Vector{Symbol})
│┌ @ abstractarray.jl:2301 Base.collect_similar(A::Vector{Symbol}, Base.Generator(f::typeof(MacroTools.totype), A::Vector{Symbol})::Base.Generator{Vector{Symbol}, typeof(MacroTools.totype)})
││┌ @ array.jl:620 Base._collect(cont::Vector{Symbol}, itr::Base.Generator{Vector{Symbol}, typeof(MacroTools.totype)}, Base.IteratorEltype(itr::Base.Generator{Vector{Symbol}, typeof(MacroTools.totype)})::Base.EltypeUnknown, Base.IteratorSize(itr::Base.Generator{Vector{Symbol}, typeof(MacroTools.totype)})::Base.HasShape{1})
│││┌ @ array.jl:710 Base.collect_to_with_first!(Base._similar_for(c::Vector{Symbol}, Base.typeof(v1::Union{Expr, Symbol})::Union{Type{Expr}, Type{Symbol}}, itr::Base.Generator{Vector{Symbol}, typeof(MacroTools.totype)}, isz::Base.HasShape{1})::Union{Vector{Expr}, Vector{Symbol}}, v1::Union{Expr, Symbol}, itr::Base.Generator{Vector{Symbol}, typeof(MacroTools.totype)}, st::Int64)
││││┌ @ array.jl:715 Base.setindex!(dest::Vector{Symbol}, v1::Expr, i1::Int64)
│││││┌ @ array.jl:853 Base.convert(_::Type{Symbol}, x::Expr)
││││││ no matching method found for call signature: Base.convert(_::Type{Symbol}, x::Expr)
│││││└────────────────
││││┌ @ array.jl:715 Base.setindex!(dest::Vector{Expr}, v1::Symbol, i1::Int64)
│││││┌ @ array.jl:853 Base.convert(_::Type{Expr}, x::Symbol)
││││││ no matching method found for call signature: Base.convert(_::Type{Expr}, x::Symbol)
│││││└────────────────
┌ @ src/utils.jl:21 Base.collect(Base.Generator(#11::MacroTools.var"#11#12", MacroTools.map(MacroTools.esc, xs::Tuple)::Any)::Base.Generator{_A, MacroTools.var"#11#12"} where _A)
│┌ @ array.jl:697 Base.collect_to_with_first!(Base._array_for(Base.typeof(v1::Expr)::Type{Expr}, Base.getproperty(itr::Base.Generator{_A, MacroTools.var"#11#12"} where _A, :iter::Symbol)::Any, isz::Any)::Any, v1::Expr, itr::Base.Generator{_A, MacroTools.var"#11#12"} where _A, st::Any)
││┌ @ array.jl:721 Base.grow_to!(dest::Any, itr::Base.Generator{_A, MacroTools.var"#11#12"} where _A, st::Any)
│││┌ @ dict.jl:153 Base.indexed_iterate(Core.getfield(Base.indexed_iterate(y::Tuple{Expr, Any}, 1)::Tuple{Expr, Int64}, 1)::Expr, 1)
││││┌ @ tuple.jl:89 Base.iterate(I::Expr)
│││││ no matching method found for call signature: Base.iterate(I::Expr)
││││└───────────────
┌ @ src/examples/destruct.jl:24 MacroTools.error("Can't destructure fields with default values")
│┌ @ error.jl:33 error(::String)
││ may throw: Base.throw($(Expr(:invoke, MethodInstance for ErrorException(::String), :(Base.ErrorException), Core.Argument(2)))::ErrorException)
│└───────────────

I used JET and fixed some actual errors and type instabilities. (this PR is the final output of my demo at our workshop)

/cc @pfitzseb Could you review this PR, please ? :)

Therefore, it is no longer safe to do funcdef[:kwargs] = map(arg->combinearg(splitarg(arg)...), funcdef[:kwargs], since this would transform a definition like f(;kwargs...) to f(;kwargs::Any...) which is no longer allowed in Julia 1.7. I don't know why that's no longer allowed, but from the REPL:

ERROR: syntax: "x::Any" is not a valid function argument name around REPL[3]:1
Stacktrace:
 [1] top-level scope
   @ REPL[3]:1

This PR setups IGNORED_SLURP_PATTERNS::Set{Symbol}, which allows users to configure patterns that should not be recognized as "slurp"s.

The motivation here is that there can be double-scores in variable names and so we sometimes may want them not to be recognized as "slurp" pattern.

For example, let's consider when we want to specify a pattern that only matches __init__() expression:

julia> @capture(:(__init__()), __init__()), @capture(:(foo()), __init__()) # you may want the latter case not to match
(true, true)

With this PR, we can configure IGNORED_SLURP_PATTERNS and achieve what we want:

julia> push!(MacroTools.IGNORED_SLURP_PATTERNS, :__init__); # configure `IGNORED_SLURP_PATTERNS`

julia> @capture(:(__init__()), __init__()), @capture(:(foo()), __init__()) # now `__init__` isn't recognized as slurp pattern
(true, false)

IGNORED_SLURP_PATTERNS is initialized as an empty set, and thus this change doesn't break any existing behavior by default.

This is causing https://github.com/JuliaLang/julia/issues/28221. At least for APIs that return a result instead of exposing MatchError, the package should use return values instead of exceptions.

MacroTools can't match macros when they appear in the type assertions in function calls. This seems like a crazy issue, but a key macro in my package depended on this functionality and now something has changed with the latest julia build and things fail. Any ideas as to why this match failure would happen?

               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.7.0-DEV.5246 (2018-05-29 13:27 UTC)
 _/ |\__'_|_|_|\__'_|  |  Commit 082be9a (8 days old master)
|__/                   |  x86_64-apple-darwin17.5.0

(v0.7) pkg> up MacroTools
  Updating registry at `~/.julia/registries/Uncurated`
  Updating git-repo `https://github.com/JuliaRegistries/Uncurated.git`
 Resolving package versions...
  Updating `~/.julia/environments/v0.7/Project.toml`
 [no changes]
  Updating `~/.julia/environments/v0.7/Manifest.toml`
 [no changes]

julia> using MacroTools

julia> x = :(foo(x::@bar(qux))).args[2].args[2]
:(#= REPL[3]:1 =# @bar qux)

julia> :(@bar(qux))
:(#= REPL[4]:1 =# @bar qux)

julia> x.head
:macrocall

julia> :(@bar(qux)).head
:macrocall

julia> x.args
3-element Array{Any,1}:
 Symbol("@bar")
 :(#= REPL[3]:1 =#)
 :qux

julia> :(@bar(qux)).args
3-element Array{Any,1}:
 Symbol("@bar")
 :(#= REPL[8]:1 =#)
 :qux

julia> @capture(x, @bar(qux))
false

julia> @capture(:(@bar(qux)), @bar(qux))
true

EDIT: It appears that this has something to do with line number nodes getting passed as arguments to macros. I rolled a custom solution for myself, but I'll dig into this sometime when I have more time. I don't think this has anything to do with the type assertions at all now.

splitdef fails with an assertion on an anonymous function with a single varargs argument:

julia> splitdef(Meta.parse(" (args...) -> 0 "))
ERROR: ArgumentError: Not a function definition: :(args...->begin
          #= none:1 =#
          0
      end)
Stacktrace:
 [1] splitdef(fdef::Expr)
   @ MacroTools ~/.julia/packages/MacroTools/PP9IQ/src/utils.jl:317

Either adding another argument or using the long form of the definition is fine:

julia> splitdef(Meta.parse(" (x, args...) -> 0 "))
Dict{Symbol, Any} with 4 entries:
  :args        => Any[:x, :(args...)]
  :body        => quote…
  :kwargs      => Any[]
  :whereparams => ()

julia> splitdef(Meta.parse(" function f(args...) 0 end "))
Dict{Symbol, Any} with 5 entries:
  :name        => :f
  :args        => Any[:(args...)]
  :kwargs      => Any[]
  :body        => quote…
  :whereparams => ()

Similar to what was noted in https://github.com/FluxML/MacroTools.jl/issues/40#issuecomment-357466077,

julia> @capture :(myvariable = 2) (lhs_Symbol = rhs_)
true

julia> @capture :(myvariable = 2) (lhs_Symbol = rhs_) | someotherpat
false

julia> @capture :(myvariable = 2) (lhs_ = rhs_) | someotherpat
true

The second match should be true as well.

This PR adds new methods splitargdef and combineargdef (in addition to existing splitarg and combinearg) that use dictionaries as an interface. The dict interface more completely expresses corner cases, as described in https://github.com/FluxML/MacroTools.jl/pull/178. In particular, it provides robust solutions for literal nothing argument defaults and avoids typeasserts when no type is given for an argument.

help?> MacroTools.isdef
  Test for function definition expressions.

julia> MacroTools.isdef(:(f() = 3))
true

julia> MacroTools.isdef(:(f()))
true

julia> MacroTools.isdef(:ix)
true

rmlines would be more useful it it worked recursively.

rmlines(Meta.parse("""
begin
   begin
      begin
end
end
end"""))
quote
    begin
        #= none:3 =#
        begin
            #= none:4 =#
        end
    end
end

Yes, one could use postwalk, but I don't think one should have to.

Also, why isn't postwalk exported?

At the very least, the documentation should manage expectations by saying that it doesn'[t work recursively and suggesting the use of postwalk.