GitLab

Yes, even though in practice in most of codegen we'll just turn around
and title-case it again for symbol export reasons.

The keywords everywhere throughout the schema language are lower-case,
and kinds are such a keyword.  They should therefore be consistently
handled and shown as lowercase.

This has bugged me for a while but it's time to fix it before any more
code starts passing by the area (and it's about to).

Some are easier than others.

I'm starting to evolve a (still ad-hoc) naming convention for the
extracted functions in order to provide some cues for how common you can
expect the thing to be.  Scalars have a lot more commonizable stuff
than non-scalars, for example.

This brings it back in line with how strings now act.

These are getting increasingly straightforward as most of the hard
problems have been ironed out already when getting the other recursives
(maps and structs) sorted.

Now, we just need to stamp out the rest of the scalars, and I think
this codegen stuff is at least early-alpha level usable!

Also in this commit: added some new error types and fixed up the
basicnode.List implementation to use it, removing some todos there.

It's oddly hard to get a test in to poke these bits -- lots of uses
funnel into shortcut codepaths that avoid a lot of the assembler.

Tests for errors on misuse (double assigment, etc) should probe this,
but we don't have any of those anywhere yet, and they're not
well-typed yet, and that's more/different than what I want to tear off
today.

Tests where the stringjoin repr'd struct is reached as a child of
other things should be another way to poke the right spot, but...
I got nervous heading that way because... well, this reason might be
wobbly, but I'm worried about keeping some order of concentrically
growing growing circles of feature sets.  I'm becoming afraid if I
ever need to do large scale updates to anything, *all* the tests will
break at once, thus giving little in the way of incrementally helpful
nudges in the direction of fixes.  If struct tests use maps, and map
tests use structs, who's depending on who, and which test failures
indicate what and where?

I think I see the outline of some heuristics that could help here;
I'll start adding more tests accordingly in subsequent commits.

Works.

Required a few fixes to the stringjoin struct representation
(namely, AssembleKey initialization hadn't been exercised yet).

Marshal is on par with basicnode.  Both basicnode and then gen stuff
does a solid job of alloc amortization on reads, so the dominant cost
remaining for both is in getting iterators.  Thus, they come out
pretty comparable overall.

Unmarshal is winning *nicely* over basicnode.  Roughly a third fewer
allocations, and gen is about 125% faster on the clock.

I haven't looked to see if unmarshal can be further improved with any
low-hanging-fruit sorts of fixes.  Wouldn't be surprised if it can.

We're gonna need more standard benchmarks... and in particular,
need them working without the marshal/unmarshal indirections.
Those are handy, but add a *lot* of noise from directions we're not
necessarily interested in when looking at different node impls.

With use of pointer here: overall benchmarks if marshalling or suchlike
are perfectly reasonable and on par with the basicnode implementations
(the biggest cost is getting iterators, which is similar in each).

With the typo causing a copy and another heap escape: that extra alloc
causes a zonking 10% slowdown on selected marshalling benchmark.

(I'm using benchmarks on the 'realgen' package.  MapNStrMap3StrInt can
be exercised now -- isn't that neat?  Suppose I'll commit that next.)

Reset methods on assemblers were needed for this.  So far, we'd gotten
along without these, because structs happen to not reuse assemblers
between fields.  But we definitely do need them in general.

It would've been possible to avoid having the reset methods on scalars,
but it doesn't seem work the effort; it would save a line of GSLOC,
but make absolutely no other practical difference, and also make the
templates a fair sight more complicated.  So meh to that.

Also, hej: now recursive maps can be compiled!  (Previously they were
stymied simply because the maybe emission hadn't been pasted in.)

Largely works, although a fix was needed on representations -- they
should *keep* returning representation nodes.

(Ironically this simplified some things, because using a method to
get the representation node means the compiler does more of the
"is this or is this not being called on a ptr" dance for us.)

These interface assertions have become pretty low-value.  Hmm.

Through compilation, at least.  No functional tests yet written.

Bunch of small fixes for maybe pointers.  (Somehow I had gotten their
embedding in the entry struct wrong, and that kinda splashed all over.)

The refactor of method names 'construct'->'fromString' also made a
bunch of the generated map code line up, since it was already written
to target that convention.

Then it's just the matter of adding the tests that compile.

And so far so good: combos of nullable values, not nullable,
and maybe with or without use of pointers all appear to compile!

Using a function name that has kind info in it feels better in many
small ways (it's a nice sanity check; if we add coercion helper
methods for more kinds, it'll sure matter; etc).

And, that method on String types belongs on the type, not just on the
representation side.

And, drop the 'NewT' method for String types; instead, have them
generate use 'pkg.Style.T.FromString' pattern.

This seems to be getting pretty close: the only compile errors left
are about not finding "fromString" methods for key reification...
which is of course sensible, because that's not what those methods
have been called yet in the cases where they've shown up;
and they also haven't been placed on type-level styles yet.

So the next commit will be a little detour to touch over those methods
formerly known as "construct" and update them a bit to this pattern.

Fixes many issues with maybes.  Some minor typo-scale stuff; some
pretty consequential.  What to do about speciated lookup methods
still has big honking questions on it: returning a maybe there if
you didn't already have data in that format can be *not cheap*.
Figuring out the best DX there is gonna be interesting.

Making progress on how to handle keys, but slow.  Complex keys are...
well, really complex.  So far I've learned several things that *won't*
work, which is fun.  It's looking unlikely that having a KeyAssembler
type at all is actually going to be a good idea.

Complex keys are a behemoth of a feature, and it's gonna take
substantial effort to sort all of that out.

(Ain't that a lovely bunch of conditions?)

Previous comment in here missed an important case.

The reuse level is currently *very* low.

Some of that is intentional -- I'm writing radically non-DRY things
on purpose in many cases, because I want to see what *actually*
repeats most often before I start trying to remove repetitions.

Some of it is not intentional; it's because I just have no idea
how to organize units of reuse when doing templating.

This latter group could improve.

\o/

schema.StructField isn't hashable (in the golang sense of the word) --
*sometimes* -- at runtime -- because the Type field can contain things
that aren't hashable.  Such as schema.TypeStruct, since it contains
a slice of schema.StructField.  lol.

This was blocking the ability to use some types in other types.

If you've been following along for a while now, you don't need to see
the benchmarks to know what's coming.  The long story short is:
allocations are the root of all evil, and we got rid of some, and now
things are significantly faster.

Here's the numbers:

basicnode (just for a baseline to compare to):

```
BenchmarkMapStrInt_3n_AssembleStandard-8         1988986               588 ns/op             520 B/op          8 allocs/op
BenchmarkMapStrInt_3n_AssembleEntry-8            2158921               559 ns/op             520 B/op          8 allocs/op
BenchmarkMapStrInt_3n_Iteration-8               19679841                67.0 ns/op            16 B/op          1 allocs/op
BenchmarkSpec_Marshal_Map3StrInt-8               1377094               870 ns/op             544 B/op          7 allocs/op
BenchmarkSpec_Marshal_Map3StrInt_CodecNull-8     4560031               278 ns/op             176 B/op          3 allocs/op
BenchmarkSpec_Unmarshal_Map3StrInt-8              368763              3239 ns/op            1608 B/op         32 allocs/op
```

realgen, previously, using fcb:

```
BenchmarkMapStrInt_3n_AssembleStandard-8         4293072               278 ns/op             208 B/op          5 allocs/op
BenchmarkMapStrInt_3n_AssembleEntry-8            4643892               259 ns/op             208 B/op          5 allocs/op
BenchmarkMapStrInt_3n_Iteration-8               20307603                59.9 ns/op            16 B/op          1 allocs/op
BenchmarkSpec_Marshal_Map3StrInt-8               1346115               913 ns/op             544 B/op          7 allocs/op
BenchmarkSpec_Marshal_Map3StrInt_CodecNull-8     4606304               256 ns/op             176 B/op          3 allocs/op
BenchmarkSpec_Unmarshal_Map3StrInt-8              425662              2793 ns/op            1160 B/op         27 allocs/op
```

realgen, new, improved:

```
BenchmarkMapStrInt_3n_AssembleStandard-8         6138765               183 ns/op             129 B/op          3 allocs/op
BenchmarkMapStrInt_3n_AssembleEntry-8            7276795               176 ns/op             129 B/op          3 allocs/op
BenchmarkMapStrInt_3n_Iteration-8               19593212                67.2 ns/op            16 B/op          1 allocs/op
BenchmarkSpec_Marshal_Map3StrInt-8               1309916               912 ns/op             544 B/op          7 allocs/op
BenchmarkSpec_Marshal_Map3StrInt_CodecNull-8     4579935               257 ns/op             176 B/op          3 allocs/op
BenchmarkSpec_Unmarshal_Map3StrInt-8              465195              2599 ns/op            1080 B/op         25 allocs/op
```

So!  About 150% improvement on assembly between gen with fcb and our new-improved no-callback system.

And about 321% improvement in total now for codegen structs over the basicnode map.

That's the kind of ratio I was looking for :)

As with all of these measurements: these will also get much bigger on bigger corpuses.
Some of the improvements here are O(n) -> O(1), and some apply even more heartily in deeper trees, etc.
But it's telling that even on very small corpuses, the impact is already huge.

Results: mixed.

The good news:

The codegen works.

We were able to wire it to the standard benchmarks (!  great success).

It is flat out faster than any other implementation to date.

The not-so-good news:

It's not _as_ fast as I wanted >:(

The strategy of using a callback ("fcb") for transmitting 'finished'
signals from child assemblers to their parents causes an allocation.

That single source of allocations turns out to be one of the most
dominant things on the pprof of the benchmark.  (And it would
absolutely be even worse if 'N' was larger than '3' -- an alloc here
shifts us from an O(1) to O(n) on fields.)

So.  Good to know!  Having end to end benchmarks is VERY exciting.

And we're going to have to go back to the drawing board on that
part involving a callback.

This diff is pretty fun.  It's our first alternative representation,
and it all works out nicely (no internal syntactical complications
encountered or any other unexpected unpleasantness).

It's also our first representation that involves potentially recursive
destructuring work over a scalar!  So, you can see the new 'construct'
method conventions appearing to handle that.  Works out neatly.
This kind of recursive destructuring happens all within the span of
processing one "token" so to speak, so naturally it can work without
any stateful machinery.

Some utility functions are added to the mixins package.
(It's sort of betraying in the name of the package, but, well,
it's an extremely expedient choice.  See comments about import
management.  tl;dr: consistency makes things easier.)

Tests for a recursive case of this will be coming soon, but requires
addressing some other design flaws with the AdjunctConfig maps.
(StructField is... not a good key.  It's not always hashable... such
as when the StructField.Type field is inhabited by TypeStruct.)
Easy enough to fix, just not going to cram it into this commit.

The way null and the 'fcb' are handled here differs from previous
generators: here, we *always* have an 'fcb', and just make a special
one for the root builder.  This is... well, it works.  It was an
attempt to simplify things and generate fewer method overrides, and
it did.  However, I'm not going to dwell on this too long, because...

The 'fcb' system is not working out well overall for other reasons:
it's causing costly allocations.  (Taking a function pointer from
a method requires at least two words: one for the function pointer
and one for the value to apply it on.  That means an allocation.)
So a serious rewrite of anything involing the 'fcb' strategy is needed.

And that is going to be a little tricky.  The 'fcb' idea was itself
a trying-slightly-too-hard-to-be-clever attempt to avoid an alternative
solution that involves generating additional types per distinct child
value slot (and I'm still unwilling to pursue that one -- it suggests
far too many types).  The next best idea I have now seems to involve a
lot of pointers into other assembler's state machines; this will surely
be a bit touchy.  But no worse than any of the rest of all this, I
suppose: it's *all* "a bit touchy".  Buckle up for fun diffs ahead.)

So, considering these 'fcb' notes, this feels a bit mixed...
"Two steps forward, one step back", so to speak.  Still: progress!
And it's very exciting that we got through several new categories of
behavior without hitting any other new roadbumps.

The label used to advance over absent optionals is a compile-time error
if not referenced, so we have to branch generation for that.

I'll want this momentarily for doing Generate from other packages.
(I might not actually commit anything relating to this for a while,
but secretly, I'm playing with it.)

Correct a half-dozen laxnesses in path handling while at it.