Porting src/RSE/UoE/jit_comparison/src/jit_comparison/python_as_metaprogramming_lang.py — but porting the point, not the code. That notebook’s whole argument was that Python is really two languages sharing one file: ordinary Python, which runs at generation time with the full ecosystem available, and Numba’s typed subset, which runs at call time with almost none of it. Metaprogramming was the discipline of keeping the seam between the two deliberate — closures, @jit, descriptors, a proxy library, all in service of writing a program that writes a program.
Julia doesn’t have that seam. There is one language, and it is its own metaprogramming language: source parses to Expr objects that are ordinary Julia values — inspectable, buildable, mutable, evaluable — and the compiler’s specialisation machinery already treats an argument’s type as a place to fix things at compile time, no closure required. This notebook shows both halves: the automatic one that comes for free, and the deliberate one (Expr, macros) for when “free” isn’t enough.
Expr:( ... ) does not run an expression, it quotes it — parses it and hands back the AST as a plain value.
Expr
Expr
head: Symbol call
args: Array{Any}((3,))
1: Symbol +
2: Symbol a
3: Expr
head: Symbol call
args: Array{Any}((3,))
1: Symbol *
2: Symbol b
3: Symbol cExpr(:call, :+, :a, Expr(:call, :*, :b, :c)): walkable and buildable with the same Expr constructor and the same field access as any other struct. eval runs it in a chosen scope:
That is the whole basis of what follows. Because code is data, “write a program that writes a program” is not a pattern bolted onto the object model — closures standing in for staged values, descriptors intercepting attribute access, a proxy library papering over the join — it is just building an Expr and handing it to the compiler.
Horner’s rule again, coefficients as a runtime Vector:
function horner_runtime(x::Vector{Float64}, coeffs::Vector{Float64})
out = similar(x)
for i in eachindex(x)
acc = 0.0
for c in coeffs
acc = acc * x[i] + c
end
out[i] = acc
end
return out
end
COEFFS = (1.0, -3.0, 2.5, 0.5, -1.25, 7.0, 0.125, -2.0)
coeffs_vec = collect(COEFFS)
x = randn(1_000_000)
horner_runtime(x, coeffs_vec); # compile, onceSame obstacle as the Numba version: length(coeffs) is a runtime value the type carries no information about, so the inner loop needs a trip-count test every iteration, and the outer loop over i cannot vectorise because LLVM cannot prove the inner loop’s shape is uniform.
In Python the fix was to move coeffs into a closure, so Numba could open the cell while compiling and treat what it found as a literal. Julia needs no closure — but it does need one nudge: a plain for c in coeffs loop over an NTuple does not unroll (Base’s tuple iterate is written to work for any length, and nothing forces it open). What does unroll is recursing on the tuple’s own structure, one dispatch per shrinking type:
horner_step(acc, xi, coeffs::Tuple{}) = acc
horner_step(acc, xi, coeffs::Tuple) =
horner_step(acc * xi + first(coeffs), xi, Base.tail(coeffs))
function horner_recursive(x::Vector{Float64}, coeffs::NTuple{N,Float64}) where {N}
out = similar(x)
for i in eachindex(x)
out[i] = horner_step(0.0, x[i], coeffs)
end
return out
end
horner_recursive(x, COEFFS); # compileNo Expr, no quote, no macro — just two ordinary method definitions. But coeffs::Tuple{} and coeffs::Tuple (a non-empty tuple) are two different types, so horner_step is not one recursive function, it is one method per tuple length, each calling a fully-resolved next method rather than looping. That resolution happens through ordinary multiple dispatch, the same mechanism behind every other Julia function call, applied to a type that happens to encode “how much work is left.” Confirm it actually unrolled and vectorised, the same LLVM-counting check the intro notebook used on the fused broadcast:
using InteractiveUtils
function fmul_vector_width(f, types)
io = IOBuffer()
code_llvm(io, f, types; debuginfo = :none)
fmuls = filter(l -> occursin("fmul", l), split(String(take!(io)), '\n'))
widths = sort(unique(m.match for l in fmuls for m in eachmatch(r"<\d+ x double>", l)))
return length(fmuls), widths
end
println(
"horner_runtime : ",
fmul_vector_width(horner_runtime, Tuple{typeof(x),typeof(coeffs_vec)}),
)
println(
"horner_recursive: ",
fmul_vector_width(horner_recursive, Tuple{typeof(x),typeof(COEFFS)}),
)horner_runtime : (2, SubString{String}[])
horner_recursive: (16, SubString{String}["<8 x double>"])horner_runtime compiles to two scalar fmuls sitting inside a loop body — one loop iteration’s worth of work, repeated at run time, on one double at a time. horner_recursive compiles to sixteen fmuls (the eight-term chain, unrolled, run twice per @time warm-up path) operating on <8 x double> — no loop over coefficients survived to be vectorised around, so LLVM was free to widen the outer loop over i instead. Same shape the closure-baked Numba kernel reached, reached here with no Expr-level metaprogramming at all — just a recursive definition whose base case is a type, not a value.
horner_step still had to be designed around dispatch from the start — fine for an algorithm written from scratch, awkward for coefficients that only exist as a literal list at a call site, with no Tuple{} base case in sight. Deliberate metaprogramming — building an Expr and handing it to the compiler yourself — is the tool for exactly that shape of problem.
A macro is a function that runs at parse time: it receives the unevaluated syntax of its arguments as Exprs (or literals, or symbols) and returns the Expr that replaces the macro call — before lowering, before types, before anything downstream ever sees the original call.
((0.0 * 2.0 + 1.0) * 2.0 + -3.0) * 2.0 + 2.5Used inside a loop, @horner disappears entirely before the function is ever lowered:
@macroexpand shows exactly what the rest of the compiler receives: not a call to some generic @horner, but the eight-deep multiply-add chain, coefficients as source-level numeric literals — indistinguishable, from here on, from a programmer having typed it out by hand.
@code_lowered shows what the dispatch-based and macro-based versions have in common downstream of two very different front ends.
horner_step, lowered:CodeInfo( 1 ─ %1 = Main.horner_step │ %2 = Main.:+ │ %3 = Main.:* │ %4 = dynamic (%3)(acc, xi) │ %5 = Main.first │ %6 = dynamic (%5)(coeffs) │ %7 = dynamic (%2)(%4, %6) │ %8 = Main.Base │ %9 = dynamic Base.getproperty(%8, :tail) │ %10 = dynamic (%9)(coeffs) │ %11 = dynamic (%1)(%7, xi, %10) └── return %11 )
horner_macro, lowered:CodeInfo( 1 ─ %1 = Main.similar │ out = (%1)(x) │ %3 = Main.eachindex │ %4 = dynamic (%3)(x) │ @_3 = Base.iterate(%4) │ %6 = @_3 │ %7 = builtin %6 === nothing │ %8 = dynamic Base.not_int(%7) └── goto #4 if not %8 2 ┄ %10 = @_3 │ i = Core.getfield(%10, 1) │ %12 = builtin Core.getfield(%10, 2) │ %13 = Main.:+ │ %14 = Main.:* │ %15 = Main.:+ │ %16 = Main.:* │ %17 = Main.:+ │ %18 = Main.:* │ %19 = Main.:+ │ %20 = Main.:* │ %21 = Main.:+ │ %22 = Main.:* │ %23 = Main.:+ │ %24 = Main.:* │ %25 = Main.:+ │ %26 = Main.:* │ %27 = Main.:+ │ %28 = Main.:* │ %29 = i │ %30 = dynamic Base.getindex(x, %29) │ %31 = dynamic (%28)(0.0, %30) │ %32 = dynamic (%27)(%31, 1.0) │ %33 = i │ %34 = dynamic Base.getindex(x, %33) │ %35 = dynamic (%26)(%32, %34) │ %36 = dynamic (%25)(%35, -3.0) │ %37 = i │ %38 = dynamic Base.getindex(x, %37) │ %39 = dynamic (%24)(%36, %38) │ %40 = dynamic (%23)(%39, 2.5) │ %41 = i │ %42 = dynamic Base.getindex(x, %41) │ %43 = dynamic (%22)(%40, %42) │ %44 = dynamic (%21)(%43, 0.5) │ %45 = i │ %46 = dynamic Base.getindex(x, %45) │ %47 = dynamic (%20)(%44, %46) │ %48 = dynamic (%19)(%47, -1.25) │ %49 = i │ %50 = dynamic Base.getindex(x, %49) │ %51 = dynamic (%18)(%48, %50) │ %52 = dynamic (%17)(%51, 7.0) │ %53 = i │ %54 = dynamic Base.getindex(x, %53) │ %55 = dynamic (%16)(%52, %54) │ %56 = dynamic (%15)(%55, 0.125) │ %57 = i │ %58 = dynamic Base.getindex(x, %57) │ %59 = dynamic (%14)(%56, %58) │ %60 = dynamic (%13)(%59, -2.0) │ %61 = out │ %62 = i │ dynamic Base.setindex!(%61, %60, %62) │ @_3 = Base.iterate(%4, %12) │ %65 = @_3 │ %66 = builtin %65 === nothing │ %67 = dynamic Base.not_int(%66) └── goto #4 if not %67 3 ─ goto #2 4 ┄ %70 = out └── return %70 )
horner_step’s lowered IR is still just a recursive call — horner_step(acc*xi + first(coeffs), xi, Base.tail(coeffs)), one method, unresolved. The unrolling has not happened yet at this stage; it depends on type inference later seeing coeffs::Tuple{} versus coeffs::Tuple as genuinely different types and resolving eight nested calls into a straight line. horner_macro’s lowered IR has no call to horner_step-like machinery at all: the macro already flattened it into eight literal multiply-adds before lowering ever ran.
Two different points in the pipeline reach the same machine code — type inference resolving a chain of dispatches, or the program rewriting its own source before the compiler even parses names into scopes. Julia offers both, in the same language, with no host/target boundary, no separate metalanguage, and no object-model tax to pay for crossing it.
The @time calls above each measured a single call — fine as a compile-vs-warm sanity check, noisy as a comparison between three implementations. BenchmarkTools.@benchmark reruns a function until the timing distribution is stable and reports the median rather than one sample; $ interpolates x, coeffs_vec and COEFFS into the benchmark expression so they are treated as constants, not slow global-variable lookups, inside the timed loop.
BenchmarkTools.Trial: 10000 samples with 1 evaluation per sample. Range (min … max): 174.977 μs … 1.097 ms ┊ GC (min … max): 0.00% … 12.02% Time (median): 419.385 μs ┊ GC (median): 0.00% Time (mean ± σ): 409.590 μs ± 126.699 μs ┊ GC (mean ± σ): 13.68% ± 14.55% ▄█ ▁ ▅▂ ▁ ▂▂▂▂▄██▇▅▃▃▃▂▂▂▂▂▂▂▁▂▂▂▂▃▄▆█▇▆▇███▇▇▆▄▃▃▂▂▂▂▂▂▇█▆▅▅▅▅▄▄▃▃▃▂▂▂ ▃ 175 μs Histogram: frequency by time 644 μs < Memory estimate: 7.63 MiB, allocs estimate: 3.
using Printf
rows = [
("runtime (Vector arg)", bench_runtime),
("recursive (NTuple dispatch)", bench_recursive),
("macro (@horner)", bench_macro),
]
baseline = median(bench_runtime).time
@printf("%-30s %12s %14s %10s\n", "implementation", "median time", "memory", "speed-up")
for (name, b) in rows
m = median(b)
@printf("%-30s %9.1f μs %11d B %8.2fx\n", name, m.time / 1e3, m.memory, baseline / m.time)
endimplementation median time memory speed-up
runtime (Vector arg) 2204.6 μs 8000072 B 1.00x
recursive (NTuple dispatch) 414.6 μs 8000072 B 5.32x
macro (@horner) 419.4 μs 8000072 B 5.26xMemory is identical across all three — one similar(x) output array, no incidental allocation — so the whole difference is in the arithmetic each generates. The recursive-dispatch and macro versions land close to each other, both several times faster than the runtime-coefficient version, for the reason the LLVM counts and the lowered IR above already made precise: neither has a loop left over coeffs for the compiler to worry about.
Python’s notebook ended with five ways the closure-and-decorator pattern leaked through Python’s object model — wrong __name__, uncallable class attributes, compiling on mere attribute access, a repr that lies, __slots__ refused — and thirty lines of descriptor-plus-proxy machinery to paper over them. None of that has an analogue here. @horner is not an object standing in for a function; it is a compiler pass that runs once, at parse time, and by the time anything could introspect horner_macro there is nothing left to distinguish it from code a person wrote by hand. Homoiconicity does not just make metaprogramming available in Julia — it removes the entire category of bookkeeping Python needed to fake it.