GitLab

... almost.  In attempting to put the sections about absent value
handling into place, I realized there's a decently sizable bug there
already.  I'm going to document this for now, move on, and come back
to it later... so it will continue to take up space in a "wip" doc.
The description of the situation is at least cleaned up now.

I spent some time poking around at other applications that I'd want to
be able to use some of this stuff on, and the lines about "minimalist"
come from that.  I suspect that in ideal outcomes, I'd like to be able
to sometimes be able to generate the type structures and native/typed
APIs for a Schema definition... but perhaps not generate all the stuff
necessary for the interface monomorphization, because perhaps I just
want an enum purely for internal program purposes; surely we should be
able to just do that.  We'll see though.  Maybe that's scope creep
as much as anything else.

All "genparts*" files are reusable bits of template;
all other "gen[A-Z](.*)" files are per kind.

Includes one code fix, as well: generated maps now return 'nil' for
the value of lookups when returning an error for not-found.
(I waffled on this for a while, because it's arguable that `ipld.Undef`
is a perfectly reasonable sentinel value for this, and it's *arguable*
that any sentinel value is better than nil.  However... no.
A much, much clearer rule is "if a method returns an error, the value
is nil", and that the contrapositive is also true: and that's what
we're now going with and sticking with.)

Less comments emitted in gen result.

Touch up the 'do not edit' comment.

Some of them still lived over the "gendemo" package, and those are now
moved over here to the proper gen package.

Linked to more of the other documents from the main HACKME doc.

A fair amount of stuff is finished.

A fair amount of stuff isn't.

These are all the type kinds and representation strategies outlined in
https://specs.ipld.io/schemas/representations.html .

This is pretty similar to the extractions that made it fly for lists.

I've added a few more comments to the "genparts*" files, because on
further reflection, a lot of those methods are... less reusable than
the name might seem to claim.  We'll see how those evolve.

Altered TestMapsContainingMaps quite a bit to now also test that
representation mode is tracked correctly across both read and creation
recursions, in a similar way to how TestListsContainingLists does
(e.g., uses some structs towards the leaf nodes, so it can use the
rename directives to create visible differences).

Correct representational creation mode still coming up, as you can
see in the fixme comment.

Tests will come in the next commit along with the creation mode.

Test for lists-of-lists and that representation-mode recursion works for both assembly and traversal.

It does.

Having a nil pointer dereference cause a panic *during* error stringing
is really unpleasant.

We shouldn't "ever" have this problem, when the library is done.
But right no we certainly do.

Would cause the same memory to get reused inappropriately; need the nil
value there to kick the child assembler to do the new allocation on the
next round.

A lot of the coments in earlier commit messages about choosing a path
and cutting through it roughly and leaving DRY for latter still
applies (see f1eeeafb).

The hardest part of writing this was giving up on having shared code
in the output.  It just can't be done along the current edges without
giving up some performance at runtime, and that's generally not a trade
we want to make.  Maybe there's different cuts for function boundaries
that would do better -- but we'll leave that for later fun work.

I did at least get a bunch of templates to be textually shared.

Comments abound in the diff for future possibilities, as well as what's
not possible to go further on without making undesired trades.

Trying to minimize gsloc here as much as feasible -- a lot of logic
*is* in common with the type-level node.

We'll see how this goes, though.  If we inspect the performance and
the assembly output at the end of this, and it's nuts, we might
reconsider how to approach this substantially.

Wanting this as I work on list representations, and making
target-of-opportunity improvements as I encounter them.

This is a really tricky thing, because we're balancing three concerns:

- minimizing our gen&template code size (and maximizing legibility)
  (okay, *four* concerns!)
- minimizing our generated output lines of code ("gsloc")
- minimizing our binary output size
- maximizing our optimizations and fine runtime speed
  (okay, *five* concerns!)

So, in addition to being the darned Spanish Inquisition about counting
the concerns here...

They're wildly divergent concerns.  And minimizing one tends to push
up the numbers for the others.  Or simply make it harder to see and
iterate on ways to manage the others.

It's likely that at some point I'm going to give up and write something
that works and seems simple as a first draft, and just roll with it.
It's important to get something more widely usable to simply be
finished enough that we can start gathering experiences on it;
the rest of this is optimization problems we can iterate and improve on
over time.

We may also need to invest in some tooling to measure things like
gsloc and binary output size, and possibly testing what's inlinable
(though simply measuring performance with regular benchmarks may remain
a decent substitute for that, even though it's limited to working on
finished systems and thus not very helpful to use in design planning).
Right now I'm running off vague intuitions from prior experiences, and
occasionally going on splunks through the disassembler, but this is a
high cost action to perform manually so I'm not doing it often and it's
not producing feedback frequently enough.  Tooling could both make it
easier to gather this feedback, and make it more maintainable over time
and available to other maintainers, which would be... capital-G Good.

It's possible these could be dedup'd... a lot, honestly.  I'm wondering
if they actually need a discrete type per kind at all, anymore; it's
seeming much more tractable to try to turn that into a "no" after all
the other successful work this morning abstracting things.

But I have some other things I want to ship and do demos and prototypes
of that's a lot more interesting than shaving more lines off today.
So just gonna roll with the existing convention and stamp these out.
Might deserve a revisit in the future.  Might not.  Time will tell.

Int is now basically all common stuff.  Now I'm really gonna quit.

Interestingly, the total line count isn't going down very fast.
108 insertions, 165 deletions, according to the git diff stat.
Evidentally the overhead of things opting into this is darn near
the amount of lines saved in many cases.

This'll probably be the end of my cleanup quest for the day, but it's
a pretty pleasing result.  More cleaned up more easily than I expected.

Decided a basic bool property for this is probably fine after all.
(For quite a while I'd been wondering if we'd end up with an enum
of variants that are more or less semantic siblings of Repr, but it
still hasn't happened, and thus it's now seeming quite unlikely.)

This one turns out to be different betweens scalars and recursives,
if in a small (and fortunately consistent) way.

Yes, even though in practice in most of codegen we'll just turn around
and title-case it again for symbol export reasons.

The keywords everywhere throughout the schema language are lower-case,
and kinds are such a keyword.  They should therefore be consistently
handled and shown as lowercase.

This has bugged me for a while but it's time to fix it before any more
code starts passing by the area (and it's about to).

Some are easier than others.

I'm starting to evolve a (still ad-hoc) naming convention for the
extracted functions in order to provide some cues for how common you can
expect the thing to be.  Scalars have a lot more commonizable stuff
than non-scalars, for example.

This brings it back in line with how strings now act.

These are getting increasingly straightforward as most of the hard
problems have been ironed out already when getting the other recursives
(maps and structs) sorted.

Now, we just need to stamp out the rest of the scalars, and I think
this codegen stuff is at least early-alpha level usable!

Also in this commit: added some new error types and fixed up the
basicnode.List implementation to use it, removing some todos there.

It's oddly hard to get a test in to poke these bits -- lots of uses
funnel into shortcut codepaths that avoid a lot of the assembler.

Tests for errors on misuse (double assigment, etc) should probe this,
but we don't have any of those anywhere yet, and they're not
well-typed yet, and that's more/different than what I want to tear off
today.

Tests where the stringjoin repr'd struct is reached as a child of
other things should be another way to poke the right spot, but...
I got nervous heading that way because... well, this reason might be
wobbly, but I'm worried about keeping some order of concentrically
growing growing circles of feature sets.  I'm becoming afraid if I
ever need to do large scale updates to anything, *all* the tests will
break at once, thus giving little in the way of incrementally helpful
nudges in the direction of fixes.  If struct tests use maps, and map
tests use structs, who's depending on who, and which test failures
indicate what and where?

I think I see the outline of some heuristics that could help here;
I'll start adding more tests accordingly in subsequent commits.

Works.