GitLab

Kinded union gen

The AssignNode method is now supplied; and the previous hacky shrug
regarding switch statements with no applicable cases also had to go.
(Turns out that the no-applicable-cases thing coincidentally worked
for the "embedall" internal implementation mode; but didn't fly for
the "interface" internal style, because that would end up with
variable declarations made but not referenced.  Harrumph.)

This template munging business got nastier as time proceded.
I think we could drastically reduce the amount of redundancy in the
arugments to `kindedUnionNodeMethodTemplateMunge` if we re-did that
function with a more structural understanding of what's going on,
and made some basic understanding of what zero values are per golang
type, etc.

That can be future work, though.  Today, I want working kinded unions,
and I just want them done enough to use; kludge tolerance high.

So far, the generation itself runs, but the result will not yet compile.
I just want a checkpoint here.  (Most of this was written a few days ago already.)

Struct tuple representation codegen

This will become relevant when we have structs with tuple representation!

Fix a somewhat horrendous parity break in struct assembly between the AssembleEntry shortcut and AssembleKey+AssembleValue.

Self-hosting gen of the schema-schema.

... With a few minor alterations.

I've taken the opportunity to make a few tweaks to naming as I went.

A few cases use keyed unions when they should be kinded; this is a
significant todo.

A few cases use keyed unions where the schema-schema declaration says
they should use inline representations!  In these, I've come to believe
the schema-schema made a mistake; we probably will update it.

The overall structure, however, is not significantly diverged.

The whole specification is still written in code: the 'SpawnFoo'
placeholder methods are in heavy use.  (This might herald the beginning
of the end, for them, though!)  (We're also still making little
hacks to dodge the current placeholder typeinfo's lack of support
for inline defns; but this is purely a problem of the placeholder
typeinfo structures, and can disappear the instant we replace them.)

If you run this generation, the emitted code is (aside from those
caveats listed above) suitable for parsing schema declarations.

...!

Which means... we can soon turn around and start using this to build
up tooling which actually uses schema JSON as a config mechanism.
Which will then bring us quite a bit closer to being able to make
free-standing usable CLI tools for working with further codegen.

There's a few other bits to go.  For starters, right now, this is
just generating output into a demo output dir.  I've made no attempt
in this commit to rig it up as a proper snake-eating-its-tail by
replacing the 'SpawnFoo' methods and placeholder type info; that'll
come in due time.  (And I think we may still have fun choices coming
up with that, incidentally; the distinction between string type names
and reified pointers is still looming, and we need to figure out what
the story is for gen outputs containing their own type descriptions
(which may touch on the same interface design choices); etc.)  I'll
probably move somewhat cautiously with this, and only cut over after
polishing the gen outputs some more... but it's now near in reach.

The size of the generated output is also very likely to need work.
We're looking something on the order of 1.6MB of generated output.
(It's *highly* redundant: if you gzip it, it's 95kb.)  Mind: I've made
*no* effort whatsoever to bring this down.  So, it's probably safe to
assume we'll find some low-hanging fruit when we actually look into it.
(I'm not yet sure what the bar will be for satisfaction with this:
I regard the current number as vaguely "seems rather high", but it's
also for a fairly sizable schema and for a lot of features provided,
so maybe some size trades are just what we're going to face in golang.)

That's it.  There will probably be some PRs to the schema-schema
documents in the specs repo shortly.  Other than those, this should
also be about ready to line up with and parse JSON output created
by the other IPLD Schema DSL->JSON parsers we already have, which
could start unlocking some really neat stuff.  🎉

Codegen: approaching self-host

This thing need to be replaced with non-placeholder code soon; it's
getting pretty nasty.

This is full of mundane plonking away at adding stars and ampersands,
of which approximately none are interesting.

(I'm particularly frustrated by this because these are all placeholder
types, and we're getting *very* close to replacing them, as we get
closer and closer to self-hosting... at which point all of this bonking
about will be made totally irrelevant.  And yet to close the last mile,
these "small" fixes are surprisingly irritating.  Ah well.)

The bits that *are* interesting:

- the Spawn functions for type info now take strings rather than types
(so that they don't provoke a cycle problem for the user when
constructing the information to describe cyclic type info);

- all of the Type info structure hold a pointer to the TypeSystem, and
use that to look up reified Type info for related types, so that their
methods don't force the caller to do that themselves.

(The TypeSystem pointer was already there, amusingly; just never before
initialized, because it hadn't turned out to be load-bearing yet.)

It also would've been possible to just change all the methods on the
Type types to return TypeName rather than full Type info.  That would
avoid the need to use a TypeSystem pointer.  I didn't because:

Overall, this was done in such a way as to minimize the diff that
impacts within the templates.  This was a goal because updating
templates is a fair bit more work than other code due to the weak
compiler support.  And we'll end up reviewing and changing these
methods when we get to our goal of self-hosting generation of the
schema types from the schema-schema, so, it's not worth pushing around
diffs in that same area when they'd be sure to be churned under soon.

These were apparently never used, or their users got churned under
at some point and I failed to clean these up at the same time.

It's a good idea to have tests that are agnostic to codegen vs
runtime type semantics implementations... but at this point we should
reengage with that by just extracting the stuff currently in the
codegen package's tests, because it's much more comprehensive and
also almost entirely abstracted already (it just happens to be in the
wrong place in the file/package hierarchy for no particular reason).

Codegen of unions, and their keyed representations

This is a clear and simple solution to the problem.

(I think I may have gotten so optimistic about type systems for a while
that I forgot that returning nils is an option.  mmmm.)

The docs on NodeBuilder already even nodded towards this possibility,
so I've just further clarified that.

After staring at this method for a while, I begin to wonder if it even
is all that useful.  To use it sensibly on structs or unions (for any
purpose other than checking if something is *not* a field/member),
you have to figure out which keys you can ask it about... which...
means you'd need the type info, to be able to enumerate that!  At which
point you'd just be able to look at the type info, and wouldn't really
need to ask the builder about ValuePrototype in this way at all,
rendering the whole thing moot.

But removing it seems a little drastic, too.  And would leave questions
about how to do those inspections on untyped things.  So.  Despite the
nibbles of oddity around this, perhaps this is still the least-bad
design that can make these situations legible.

I can't quite claim tests passed on the *first* shot... but,
the first shot after mostly syntactical (rather than semantic) fixes?
Yeah, actually.  That was pretty fun.

Snuck in a bit of DRY'ing up.  The repetition in BeginMap methods
got to me, and was low hanging fruit, so I extract that from unions
and also backported it to structs.

Errors got some work in this commit, because it turns out I've
straightjacketed myself by not allowing "fmt.Errorf" due to imports.
There's a lot to do there, and I only tackled what was directly
critical to get this commit about unions across the finish line,
but there's a few remarks in comments about where to go next.

Some more comments about future work on the type info holder types
also appears; I'm starting to skid on those placeholders and their
issues more and more.  I really hope we can get to replace those
sooner than later.

And... also, yes, the idea of not having a "focus" state field in
assemblers really bit it, hard, as speculated in the previous
commit message.  I ended up using 'ca' in more switches than I
expected, simply because it's easier to use that than have the
conditonal templating branches that would be necessary to use the other
tagging mechanisms that do also have sufficient info.

One big fixme in the core interfaces for nodebuilders (wince):
the ValuePrototype method can error sometimes, and that hasn't been
accounted for.  Need to make a decision about what to do there.
It's not really an exercised path in practice, but it shouldn't
contain caltrops, regardless of how frequently used it is.
(This probably would've come up earlier, except there's a bunch of
stubs about ValuePrototype in other parts of codegen already; all of
them need backfill, but haven't yet made it to top of the todo heap.)

But despite all the fixmes accumulated, this does bring unions
to be a usable thing, and definitively proves out that the design
still works, even for what turns out to be one of the most complicated
parts of the schema system!  It's very, *very* exciting to add the
checkmarks to this part of the feature table -- it's one of the places
I most feared "unknown unknowns", now it's put to bed.  Woot!

... for the type-level assemblers, anyway.

We *still* can't do e2e tests on this that include compiling the
generated code; not until we get at least one representation strategy
implemented.  All the templates exercise, though (even though you
might have to uncomment some of the test "skip" lines to do it).

I'll move on to doing the keyed representation next, and that will
be the thing we use to prove out all of this stuff.  It should be
the easiest one to do now, too, since it's semantically very very
similar to how we made the type-level behaviors work.

These are turning out to be *very* fun.

In the process of detailing how child assemblers work for unions,
it has become time to consider how separately-allocated child
assemblers are handled, and what their existence implies about
the rest of our logical rules how all the assemblers behave in concert.
Like so many things in this endeavor... the far-reachingness of
the implications of even seemingly very mild choices is... high.

I think it's quite likely that the current claims about union
assemblers not needing extra state for set/focus might turn out to be
redacted in upcoming commits, and for the strangest reason: we might
actually need that state to be persisted (outside of 'w') so that
the *reset* mechanism, of all things, can be efficient.
We'll see; still sort of tumbling that thought around in my mind.
There are various options.

Outline for keyed representation, but only enough to get some more
things compilable and start getting all the templates exercised.

Still can't really e2e test the things we've got without getting
the rest of it and its builders finished.  It's coming along pretty
reasonably, though, it seems.

Generates the type structure itself, and a marker interface.
Not much else yet.

Wired test harnesses and plugged out one example.

It's a bummer we don't really have a great way to poke part of it into
running until the whole thing satisfies the generator interface.
But it'll come soon enough.

Golang templates are mostly pretty smooth to get started with,
but this particular issue is really a bizarre tripwire, imo.

Will clean up the existing "range modifies dot unhelpfully" messes
sometime over the rainbow; I just refuse to make more of them today.

Also make some of the placeholder construction functions.
(Not all of them.  Just the ones I intend to use first.)

There was some very unfinished placeholder stuff going on there.
(And ironically, it wasn't written in a very clear union-like way;
it gets a lot less icky when it's rewritten so that it's uniony.)

The appearance of the word "any" there has started to perturb me;
"any" is a concept that we also need to describe in schemas, and that
type has nothing to do with it.  It's more of a base mixin.

It's kind of dark.  I've been leaving remarks strewn around github
for a while now to the general effect of "please please please
prefer keyed unions whenever you can; the others can be Not Fast".
But it's even more nasty in terms of details like error handling
than I think I had yet realized.  Sigh.

Oh well.  Remember: we exist to describe data that people *have made*,
not just the data that we want to encourage them to make in the future.

Expect this to evolve slowly over and the course of *many* commits.
Unions are exceptionally tricky, and their various representations
are by far the most complicated parts of IPLD Schemas.

In particular, the fact that how to interpret data *inside* a union
may vary, whilst some representations may not actually tell us which
union member we have until later in the stream... is going to produce
a lot of absolutely fabulous complexity.  Look forward to it.