GitLab

Correct representational creation mode still coming up, as you can
see in the fixme comment.

Tests will come in the next commit along with the creation mode.

Test for lists-of-lists and that representation-mode recursion works for both assembly and traversal.

It does.

Having a nil pointer dereference cause a panic *during* error stringing
is really unpleasant.

We shouldn't "ever" have this problem, when the library is done.
But right no we certainly do.

Would cause the same memory to get reused inappropriately; need the nil
value there to kick the child assembler to do the new allocation on the
next round.

A lot of the coments in earlier commit messages about choosing a path
and cutting through it roughly and leaving DRY for latter still
applies (see f1eeeafb).

The hardest part of writing this was giving up on having shared code
in the output.  It just can't be done along the current edges without
giving up some performance at runtime, and that's generally not a trade
we want to make.  Maybe there's different cuts for function boundaries
that would do better -- but we'll leave that for later fun work.

I did at least get a bunch of templates to be textually shared.

Comments abound in the diff for future possibilities, as well as what's
not possible to go further on without making undesired trades.

Trying to minimize gsloc here as much as feasible -- a lot of logic
*is* in common with the type-level node.

We'll see how this goes, though.  If we inspect the performance and
the assembly output at the end of this, and it's nuts, we might
reconsider how to approach this substantially.

Wanting this as I work on list representations, and making
target-of-opportunity improvements as I encounter them.

This is a really tricky thing, because we're balancing three concerns:

- minimizing our gen&template code size (and maximizing legibility)
  (okay, *four* concerns!)
- minimizing our generated output lines of code ("gsloc")
- minimizing our binary output size
- maximizing our optimizations and fine runtime speed
  (okay, *five* concerns!)

So, in addition to being the darned Spanish Inquisition about counting
the concerns here...

They're wildly divergent concerns.  And minimizing one tends to push
up the numbers for the others.  Or simply make it harder to see and
iterate on ways to manage the others.

It's likely that at some point I'm going to give up and write something
that works and seems simple as a first draft, and just roll with it.
It's important to get something more widely usable to simply be
finished enough that we can start gathering experiences on it;
the rest of this is optimization problems we can iterate and improve on
over time.

We may also need to invest in some tooling to measure things like
gsloc and binary output size, and possibly testing what's inlinable
(though simply measuring performance with regular benchmarks may remain
a decent substitute for that, even though it's limited to working on
finished systems and thus not very helpful to use in design planning).
Right now I'm running off vague intuitions from prior experiences, and
occasionally going on splunks through the disassembler, but this is a
high cost action to perform manually so I'm not doing it often and it's
not producing feedback frequently enough.  Tooling could both make it
easier to gather this feedback, and make it more maintainable over time
and available to other maintainers, which would be... capital-G Good.

It's possible these could be dedup'd... a lot, honestly.  I'm wondering
if they actually need a discrete type per kind at all, anymore; it's
seeming much more tractable to try to turn that into a "no" after all
the other successful work this morning abstracting things.

But I have some other things I want to ship and do demos and prototypes
of that's a lot more interesting than shaving more lines off today.
So just gonna roll with the existing convention and stamp these out.
Might deserve a revisit in the future.  Might not.  Time will tell.

Int is now basically all common stuff.  Now I'm really gonna quit.

Interestingly, the total line count isn't going down very fast.
108 insertions, 165 deletions, according to the git diff stat.
Evidentally the overhead of things opting into this is darn near
the amount of lines saved in many cases.

This'll probably be the end of my cleanup quest for the day, but it's
a pretty pleasing result.  More cleaned up more easily than I expected.

Decided a basic bool property for this is probably fine after all.
(For quite a while I'd been wondering if we'd end up with an enum
of variants that are more or less semantic siblings of Repr, but it
still hasn't happened, and thus it's now seeming quite unlikely.)

This one turns out to be different betweens scalars and recursives,
if in a small (and fortunately consistent) way.

Yes, even though in practice in most of codegen we'll just turn around
and title-case it again for symbol export reasons.

The keywords everywhere throughout the schema language are lower-case,
and kinds are such a keyword.  They should therefore be consistently
handled and shown as lowercase.

This has bugged me for a while but it's time to fix it before any more
code starts passing by the area (and it's about to).

Some are easier than others.

I'm starting to evolve a (still ad-hoc) naming convention for the
extracted functions in order to provide some cues for how common you can
expect the thing to be.  Scalars have a lot more commonizable stuff
than non-scalars, for example.

This brings it back in line with how strings now act.

These are getting increasingly straightforward as most of the hard
problems have been ironed out already when getting the other recursives
(maps and structs) sorted.

Now, we just need to stamp out the rest of the scalars, and I think
this codegen stuff is at least early-alpha level usable!

Also in this commit: added some new error types and fixed up the
basicnode.List implementation to use it, removing some todos there.

It's oddly hard to get a test in to poke these bits -- lots of uses
funnel into shortcut codepaths that avoid a lot of the assembler.

Tests for errors on misuse (double assigment, etc) should probe this,
but we don't have any of those anywhere yet, and they're not
well-typed yet, and that's more/different than what I want to tear off
today.

Tests where the stringjoin repr'd struct is reached as a child of
other things should be another way to poke the right spot, but...
I got nervous heading that way because... well, this reason might be
wobbly, but I'm worried about keeping some order of concentrically
growing growing circles of feature sets.  I'm becoming afraid if I
ever need to do large scale updates to anything, *all* the tests will
break at once, thus giving little in the way of incrementally helpful
nudges in the direction of fixes.  If struct tests use maps, and map
tests use structs, who's depending on who, and which test failures
indicate what and where?

I think I see the outline of some heuristics that could help here;
I'll start adding more tests accordingly in subsequent commits.

Works.

Required a few fixes to the stringjoin struct representation
(namely, AssembleKey initialization hadn't been exercised yet).

Marshal is on par with basicnode.  Both basicnode and then gen stuff
does a solid job of alloc amortization on reads, so the dominant cost
remaining for both is in getting iterators.  Thus, they come out
pretty comparable overall.

Unmarshal is winning *nicely* over basicnode.  Roughly a third fewer
allocations, and gen is about 125% faster on the clock.

I haven't looked to see if unmarshal can be further improved with any
low-hanging-fruit sorts of fixes.  Wouldn't be surprised if it can.

We're gonna need more standard benchmarks... and in particular,
need them working without the marshal/unmarshal indirections.
Those are handy, but add a *lot* of noise from directions we're not
necessarily interested in when looking at different node impls.

With use of pointer here: overall benchmarks if marshalling or suchlike
are perfectly reasonable and on par with the basicnode implementations
(the biggest cost is getting iterators, which is similar in each).

With the typo causing a copy and another heap escape: that extra alloc
causes a zonking 10% slowdown on selected marshalling benchmark.

(I'm using benchmarks on the 'realgen' package.  MapNStrMap3StrInt can
be exercised now -- isn't that neat?  Suppose I'll commit that next.)

Reset methods on assemblers were needed for this.  So far, we'd gotten
along without these, because structs happen to not reuse assemblers
between fields.  But we definitely do need them in general.

It would've been possible to avoid having the reset methods on scalars,
but it doesn't seem work the effort; it would save a line of GSLOC,
but make absolutely no other practical difference, and also make the
templates a fair sight more complicated.  So meh to that.

Also, hej: now recursive maps can be compiled!  (Previously they were
stymied simply because the maybe emission hadn't been pasted in.)

Largely works, although a fix was needed on representations -- they
should *keep* returning representation nodes.

(Ironically this simplified some things, because using a method to
get the representation node means the compiler does more of the
"is this or is this not being called on a ptr" dance for us.)

These interface assertions have become pretty low-value.  Hmm.

Through compilation, at least.  No functional tests yet written.

Bunch of small fixes for maybe pointers.  (Somehow I had gotten their
embedding in the entry struct wrong, and that kinda splashed all over.)

The refactor of method names 'construct'->'fromString' also made a
bunch of the generated map code line up, since it was already written
to target that convention.

Then it's just the matter of adding the tests that compile.

And so far so good: combos of nullable values, not nullable,
and maybe with or without use of pointers all appear to compile!

Using a function name that has kind info in it feels better in many
small ways (it's a nice sanity check; if we add coercion helper
methods for more kinds, it'll sure matter; etc).

And, that method on String types belongs on the type, not just on the
representation side.

And, drop the 'NewT' method for String types; instead, have them
generate use 'pkg.Style.T.FromString' pattern.

This seems to be getting pretty close: the only compile errors left
are about not finding "fromString" methods for key reification...
which is of course sensible, because that's not what those methods
have been called yet in the cases where they've shown up;
and they also haven't been placed on type-level styles yet.

So the next commit will be a little detour to touch over those methods
formerly known as "construct" and update them a bit to this pattern.

Fixes many issues with maybes.  Some minor typo-scale stuff; some
pretty consequential.  What to do about speciated lookup methods
still has big honking questions on it: returning a maybe there if
you didn't already have data in that format can be *not cheap*.
Figuring out the best DX there is gonna be interesting.

Making progress on how to handle keys, but slow.  Complex keys are...
well, really complex.  So far I've learned several things that *won't*
work, which is fun.  It's looking unlikely that having a KeyAssembler
type at all is actually going to be a good idea.