GitLab

Previously, it was manually written prototypes of what gen "would" look like.

Now it's the real deal :3

Going to move it over to replace the (currently hand-written) gendemo
package shortly... but spread that over a few commits, in case the
diffs turn out interesting to look at.

Less comments emitted in gen result.

Touch up the 'do not edit' comment.

Some of them still lived over the "gendemo" package, and those are now
moved over here to the proper gen package.

Linked to more of the other documents from the main HACKME doc.

A fair amount of stuff is finished.

A fair amount of stuff isn't.

These are all the type kinds and representation strategies outlined in
https://specs.ipld.io/schemas/representations.html .

This is pretty similar to the extractions that made it fly for lists.

I've added a few more comments to the "genparts*" files, because on
further reflection, a lot of those methods are... less reusable than
the name might seem to claim.  We'll see how those evolve.

Altered TestMapsContainingMaps quite a bit to now also test that
representation mode is tracked correctly across both read and creation
recursions, in a similar way to how TestListsContainingLists does
(e.g., uses some structs towards the leaf nodes, so it can use the
rename directives to create visible differences).

Correct representational creation mode still coming up, as you can
see in the fixme comment.

Tests will come in the next commit along with the creation mode.

Test for lists-of-lists and that representation-mode recursion works for both assembly and traversal.

It does.

Having a nil pointer dereference cause a panic *during* error stringing
is really unpleasant.

We shouldn't "ever" have this problem, when the library is done.
But right no we certainly do.

Would cause the same memory to get reused inappropriately; need the nil
value there to kick the child assembler to do the new allocation on the
next round.

A lot of the coments in earlier commit messages about choosing a path
and cutting through it roughly and leaving DRY for latter still
applies (see f1eeeafb).

The hardest part of writing this was giving up on having shared code
in the output.  It just can't be done along the current edges without
giving up some performance at runtime, and that's generally not a trade
we want to make.  Maybe there's different cuts for function boundaries
that would do better -- but we'll leave that for later fun work.

I did at least get a bunch of templates to be textually shared.

Comments abound in the diff for future possibilities, as well as what's
not possible to go further on without making undesired trades.

Trying to minimize gsloc here as much as feasible -- a lot of logic
*is* in common with the type-level node.

We'll see how this goes, though.  If we inspect the performance and
the assembly output at the end of this, and it's nuts, we might
reconsider how to approach this substantially.

Wanting this as I work on list representations, and making
target-of-opportunity improvements as I encounter them.

This is a really tricky thing, because we're balancing three concerns:

- minimizing our gen&template code size (and maximizing legibility)
  (okay, *four* concerns!)
- minimizing our generated output lines of code ("gsloc")
- minimizing our binary output size
- maximizing our optimizations and fine runtime speed
  (okay, *five* concerns!)

So, in addition to being the darned Spanish Inquisition about counting
the concerns here...

They're wildly divergent concerns.  And minimizing one tends to push
up the numbers for the others.  Or simply make it harder to see and
iterate on ways to manage the others.

It's likely that at some point I'm going to give up and write something
that works and seems simple as a first draft, and just roll with it.
It's important to get something more widely usable to simply be
finished enough that we can start gathering experiences on it;
the rest of this is optimization problems we can iterate and improve on
over time.

We may also need to invest in some tooling to measure things like
gsloc and binary output size, and possibly testing what's inlinable
(though simply measuring performance with regular benchmarks may remain
a decent substitute for that, even though it's limited to working on
finished systems and thus not very helpful to use in design planning).
Right now I'm running off vague intuitions from prior experiences, and
occasionally going on splunks through the disassembler, but this is a
high cost action to perform manually so I'm not doing it often and it's
not producing feedback frequently enough.  Tooling could both make it
easier to gather this feedback, and make it more maintainable over time
and available to other maintainers, which would be... capital-G Good.

It's possible these could be dedup'd... a lot, honestly.  I'm wondering
if they actually need a discrete type per kind at all, anymore; it's
seeming much more tractable to try to turn that into a "no" after all
the other successful work this morning abstracting things.

But I have some other things I want to ship and do demos and prototypes
of that's a lot more interesting than shaving more lines off today.
So just gonna roll with the existing convention and stamp these out.
Might deserve a revisit in the future.  Might not.  Time will tell.

Int is now basically all common stuff.  Now I'm really gonna quit.

Interestingly, the total line count isn't going down very fast.
108 insertions, 165 deletions, according to the git diff stat.
Evidentally the overhead of things opting into this is darn near
the amount of lines saved in many cases.

This'll probably be the end of my cleanup quest for the day, but it's
a pretty pleasing result.  More cleaned up more easily than I expected.

Decided a basic bool property for this is probably fine after all.
(For quite a while I'd been wondering if we'd end up with an enum
of variants that are more or less semantic siblings of Repr, but it
still hasn't happened, and thus it's now seeming quite unlikely.)

This one turns out to be different betweens scalars and recursives,
if in a small (and fortunately consistent) way.

Yes, even though in practice in most of codegen we'll just turn around
and title-case it again for symbol export reasons.

The keywords everywhere throughout the schema language are lower-case,
and kinds are such a keyword.  They should therefore be consistently
handled and shown as lowercase.

This has bugged me for a while but it's time to fix it before any more
code starts passing by the area (and it's about to).

Some are easier than others.

I'm starting to evolve a (still ad-hoc) naming convention for the
extracted functions in order to provide some cues for how common you can
expect the thing to be.  Scalars have a lot more commonizable stuff
than non-scalars, for example.

This brings it back in line with how strings now act.

These are getting increasingly straightforward as most of the hard
problems have been ironed out already when getting the other recursives
(maps and structs) sorted.

Now, we just need to stamp out the rest of the scalars, and I think
this codegen stuff is at least early-alpha level usable!

Also in this commit: added some new error types and fixed up the
basicnode.List implementation to use it, removing some todos there.

It's oddly hard to get a test in to poke these bits -- lots of uses
funnel into shortcut codepaths that avoid a lot of the assembler.

Tests for errors on misuse (double assigment, etc) should probe this,
but we don't have any of those anywhere yet, and they're not
well-typed yet, and that's more/different than what I want to tear off
today.

Tests where the stringjoin repr'd struct is reached as a child of
other things should be another way to poke the right spot, but...
I got nervous heading that way because... well, this reason might be
wobbly, but I'm worried about keeping some order of concentrically
growing growing circles of feature sets.  I'm becoming afraid if I
ever need to do large scale updates to anything, *all* the tests will
break at once, thus giving little in the way of incrementally helpful
nudges in the direction of fixes.  If struct tests use maps, and map
tests use structs, who's depending on who, and which test failures
indicate what and where?

I think I see the outline of some heuristics that could help here;
I'll start adding more tests accordingly in subsequent commits.

Works.

Required a few fixes to the stringjoin struct representation
(namely, AssembleKey initialization hadn't been exercised yet).

Marshal is on par with basicnode.  Both basicnode and then gen stuff
does a solid job of alloc amortization on reads, so the dominant cost
remaining for both is in getting iterators.  Thus, they come out
pretty comparable overall.

Unmarshal is winning *nicely* over basicnode.  Roughly a third fewer
allocations, and gen is about 125% faster on the clock.

I haven't looked to see if unmarshal can be further improved with any
low-hanging-fruit sorts of fixes.  Wouldn't be surprised if it can.

We're gonna need more standard benchmarks... and in particular,
need them working without the marshal/unmarshal indirections.
Those are handy, but add a *lot* of noise from directions we're not
necessarily interested in when looking at different node impls.

With use of pointer here: overall benchmarks if marshalling or suchlike
are perfectly reasonable and on par with the basicnode implementations
(the biggest cost is getting iterators, which is similar in each).

With the typo causing a copy and another heap escape: that extra alloc
causes a zonking 10% slowdown on selected marshalling benchmark.

(I'm using benchmarks on the 'realgen' package.  MapNStrMap3StrInt can
be exercised now -- isn't that neat?  Suppose I'll commit that next.)

Reset methods on assemblers were needed for this.  So far, we'd gotten
along without these, because structs happen to not reuse assemblers
between fields.  But we definitely do need them in general.

It would've been possible to avoid having the reset methods on scalars,
but it doesn't seem work the effort; it would save a line of GSLOC,
but make absolutely no other practical difference, and also make the
templates a fair sight more complicated.  So meh to that.