GitLab

Outline for keyed representation, but only enough to get some more
things compilable and start getting all the templates exercised.

Still can't really e2e test the things we've got without getting
the rest of it and its builders finished.  It's coming along pretty
reasonably, though, it seems.

Generates the type structure itself, and a marker interface.
Not much else yet.

Wired test harnesses and plugged out one example.

It's a bummer we don't really have a great way to poke part of it into
running until the whole thing satisfies the generator interface.
But it'll come soon enough.

Golang templates are mostly pretty smooth to get started with,
but this particular issue is really a bizarre tripwire, imo.

Will clean up the existing "range modifies dot unhelpfully" messes
sometime over the rainbow; I just refuse to make more of them today.

It's kind of dark.  I've been leaving remarks strewn around github
for a while now to the general effect of "please please please
prefer keyed unions whenever you can; the others can be Not Fast".
But it's even more nasty in terms of details like error handling
than I think I had yet realized.  Sigh.

Oh well.  Remember: we exist to describe data that people *have made*,
not just the data that we want to encourage them to make in the future.

Expect this to evolve slowly over and the course of *many* commits.
Unions are exceptionally tricky, and their various representations
are by far the most complicated parts of IPLD Schemas.

In particular, the fact that how to interpret data *inside* a union
may vary, whilst some representations may not actually tell us which
union member we have until later in the stream... is going to produce
a lot of absolutely fabulous complexity.  Look forward to it.

Hopefully this increases clarity and eases comprehension.

Notes and discussion can be found at
https://github.com/ipld/go-ipld-prime/issues/54 (and also
I suppose in some of our weekly video chats, but I'd have
to go on quite a dig to find the relevant links and time).

Many many refernces to 'ns' are also updated to 'np',
making the line count in this diff pretty wild.

See the changelog for discussion; this had already been on the docket
for a while now.

... and it will further become LookupByNode shortly, but that will be
a separate commit.

See the changelog for discussion; this had already been on the docket
for a while now.

... almost.  In attempting to put the sections about absent value
handling into place, I realized there's a decently sizable bug there
already.  I'm going to document this for now, move on, and come back
to it later... so it will continue to take up space in a "wip" doc.
The description of the situation is at least cleaned up now.

I spent some time poking around at other applications that I'd want to
be able to use some of this stuff on, and the lines about "minimalist"
come from that.  I suspect that in ideal outcomes, I'd like to be able
to sometimes be able to generate the type structures and native/typed
APIs for a Schema definition... but perhaps not generate all the stuff
necessary for the interface monomorphization, because perhaps I just
want an enum purely for internal program purposes; surely we should be
able to just do that.  We'll see though.  Maybe that's scope creep
as much as anything else.

All "genparts*" files are reusable bits of template;
all other "gen[A-Z](.*)" files are per kind.

Includes one code fix, as well: generated maps now return 'nil' for
the value of lookups when returning an error for not-found.
(I waffled on this for a while, because it's arguable that `ipld.Undef`
is a perfectly reasonable sentinel value for this, and it's *arguable*
that any sentinel value is better than nil.  However... no.
A much, much clearer rule is "if a method returns an error, the value
is nil", and that the contrapositive is also true: and that's what
we're now going with and sticking with.)

Less comments emitted in gen result.

Touch up the 'do not edit' comment.

Some of them still lived over the "gendemo" package, and those are now
moved over here to the proper gen package.

Linked to more of the other documents from the main HACKME doc.

A fair amount of stuff is finished.

A fair amount of stuff isn't.

These are all the type kinds and representation strategies outlined in
https://specs.ipld.io/schemas/representations.html .

This is pretty similar to the extractions that made it fly for lists.

I've added a few more comments to the "genparts*" files, because on
further reflection, a lot of those methods are... less reusable than
the name might seem to claim.  We'll see how those evolve.

Altered TestMapsContainingMaps quite a bit to now also test that
representation mode is tracked correctly across both read and creation
recursions, in a similar way to how TestListsContainingLists does
(e.g., uses some structs towards the leaf nodes, so it can use the
rename directives to create visible differences).

Correct representational creation mode still coming up, as you can
see in the fixme comment.

Tests will come in the next commit along with the creation mode.

Test for lists-of-lists and that representation-mode recursion works for both assembly and traversal.

It does.

Would cause the same memory to get reused inappropriately; need the nil
value there to kick the child assembler to do the new allocation on the
next round.

A lot of the coments in earlier commit messages about choosing a path
and cutting through it roughly and leaving DRY for latter still
applies (see f1eeeafb).

The hardest part of writing this was giving up on having shared code
in the output.  It just can't be done along the current edges without
giving up some performance at runtime, and that's generally not a trade
we want to make.  Maybe there's different cuts for function boundaries
that would do better -- but we'll leave that for later fun work.

I did at least get a bunch of templates to be textually shared.

Comments abound in the diff for future possibilities, as well as what's
not possible to go further on without making undesired trades.

Trying to minimize gsloc here as much as feasible -- a lot of logic
*is* in common with the type-level node.

We'll see how this goes, though.  If we inspect the performance and
the assembly output at the end of this, and it's nuts, we might
reconsider how to approach this substantially.

Wanting this as I work on list representations, and making
target-of-opportunity improvements as I encounter them.

This is a really tricky thing, because we're balancing three concerns:

- minimizing our gen&template code size (and maximizing legibility)
  (okay, *four* concerns!)
- minimizing our generated output lines of code ("gsloc")
- minimizing our binary output size
- maximizing our optimizations and fine runtime speed
  (okay, *five* concerns!)

So, in addition to being the darned Spanish Inquisition about counting
the concerns here...

They're wildly divergent concerns.  And minimizing one tends to push
up the numbers for the others.  Or simply make it harder to see and
iterate on ways to manage the others.

It's likely that at some point I'm going to give up and write something
that works and seems simple as a first draft, and just roll with it.
It's important to get something more widely usable to simply be
finished enough that we can start gathering experiences on it;
the rest of this is optimization problems we can iterate and improve on
over time.

We may also need to invest in some tooling to measure things like
gsloc and binary output size, and possibly testing what's inlinable
(though simply measuring performance with regular benchmarks may remain
a decent substitute for that, even though it's limited to working on
finished systems and thus not very helpful to use in design planning).
Right now I'm running off vague intuitions from prior experiences, and
occasionally going on splunks through the disassembler, but this is a
high cost action to perform manually so I'm not doing it often and it's
not producing feedback frequently enough.  Tooling could both make it
easier to gather this feedback, and make it more maintainable over time
and available to other maintainers, which would be... capital-G Good.

It's possible these could be dedup'd... a lot, honestly.  I'm wondering
if they actually need a discrete type per kind at all, anymore; it's
seeming much more tractable to try to turn that into a "no" after all
the other successful work this morning abstracting things.

But I have some other things I want to ship and do demos and prototypes
of that's a lot more interesting than shaving more lines off today.
So just gonna roll with the existing convention and stamp these out.
Might deserve a revisit in the future.  Might not.  Time will tell.

Int is now basically all common stuff.  Now I'm really gonna quit.

Interestingly, the total line count isn't going down very fast.
108 insertions, 165 deletions, according to the git diff stat.
Evidentally the overhead of things opting into this is darn near
the amount of lines saved in many cases.

This'll probably be the end of my cleanup quest for the day, but it's
a pretty pleasing result.  More cleaned up more easily than I expected.

Decided a basic bool property for this is probably fine after all.
(For quite a while I'd been wondering if we'd end up with an enum
of variants that are more or less semantic siblings of Repr, but it
still hasn't happened, and thus it's now seeming quite unlikely.)

This one turns out to be different betweens scalars and recursives,
if in a small (and fortunately consistent) way.

Yes, even though in practice in most of codegen we'll just turn around
and title-case it again for symbol export reasons.

The keywords everywhere throughout the schema language are lower-case,
and kinds are such a keyword.  They should therefore be consistently
handled and shown as lowercase.

This has bugged me for a while but it's time to fix it before any more
code starts passing by the area (and it's about to).

Some are easier than others.

I'm starting to evolve a (still ad-hoc) naming convention for the
extracted functions in order to provide some cues for how common you can
expect the thing to be.  Scalars have a lot more commonizable stuff
than non-scalars, for example.