GitLab

Merge-ignore branch 'schema-dmt-unification'

This commit is a merge commit, but using the "ours" strategy -- meaning
that the contents are functionally ignored and have no impact on the
branch they're being merged into.

Something went off the rails at some point here, and I'm not sure what,
but I'm calling it quits and this work will have to be rebooted in
a fresh set of patches sometime in the future.

The "hackme" file discussing the choices: probably salvagable.
It's not perfect (some sections read harsher than others; should redo
it with a table that applies the same checklist to each option),
and obviously the final conclusion it came to is questionable, but
most of the discussion is probably good.

The "carrier types" sidequest: extremely questionable.
The micro-codegenerator I made for that lives on in its own repo:
see https://github.com/warpfork/go-quickimmut .  But overall,
I don't think this went well.  With these, we got immutability,
and we got it without an import cycle or any direct dependence
on the IPLD Schema codegen output, but... we lost other things,
like the ability to differentiate empty lists from nil lists,
and it drops map ordering again, etc.  All of these are problems
solved by the IPLD Schema codegen, and losing those features again
was just... painful.  And the more places we're stuck flipping
between various semantics for representing "Maybe", the vastly
more likely it becomes to be farming bugs; this approach hit that.
Tracking "Is this nil or empty at this phase of its life" got
very confusing and is one of the main reasons I'm feeling it's
probably wise to put this whole thing down.

The first time I re-wrote a custom order-preservation feature,
I was like "ugh, golang, but okay, fine".  The second and third
and fourth times I ran into it, and realized not doing the work
again would result in randomized error message orders and muck up
my attempts at unit tests for error paths... I was less happy.
I still don't know what the solution to this is, other than trying
to use the IPLD Schema codegen in a cyclic way, because it *does*
solve these problems.

All of the transformation code in the dmt package which flipped
things into the compiler structures: awful.
To be fair, I expected that to be awful.  But it was really awful.

The "carrier types" stuff all being attached to a Compiler type,
for sheer grouping: extremely questionable.
If we were going to have this architecture of types|compiler|dmt
overall, I'd hands down absolutely full-stop definitely no-question
want the compiler to just be its own package.  Unfortunately,
as you already know, golang: if we want the types metadata types to
be immutable, all the code for building them has to be in-package.
Any form of namespacing at all would be an adequate substitute for
a full-on package boundary.  Possibly even just a feature that allowed
some things in a package to be grouped together in the *documentation*
would be satiating!

The "compiler_rules.go" file: really quite good!
Not entirely certain of the "first rule in a set to flunk causes
short-circuit" tactic, because there's at least one case where that
resulted in under-informative error messages that could've reasonably
identified two issues in one pass if not for the short-circuit.
But otherwise, the table-driven approach seemed promising.

Holistically: it seemed like having a "compiler" system that was
separate from the "dmt" data holder types would be a nice separation
of concerns.  When actually writing it: no, it was not.  The compiler
system ended up needing to pass through almost the exact range of
semantics that the dmt expresses (whether good or bad), so that when
the validator system was built on the compiler's data types, it could
provide reasonable responses to any issues in the data that might've
originated from the dmt format (and in turn this ultimately matters
for end-user experience).  With that degree of coupling, the compiler
system ends up forced to have a very limited and sharp-edged API
that's not at all natural, and certainly doesn't add any value.
It's possible this is inevitable (we didn't start this quest in order
to get a nice golang API, we started it to get rid of a dang import
cycle!), but it definitely generated pain.

I'm frankly not sure what the path forward here is.  Having the
validation logic attach to the dmt would solve the coupling pressure;
but the tradeoff would be there's no validation logic on the
constructors which produce the in-memory type info!  Is that okay?
I dunno.  It would definitely make me grit my teeth, but maybe it's one
of the least bad options left, seeing as how many other angles of
attack seem to have turned pretty sour now.
Another angle that deserves more thought is cyclebreaking by removing
self-description from generated types (this gets a mention in the
hackme doc in the diff already, but perhaps isn't studied far enough).

Whatever it is: it's going to start as a new body of work.
This well here is dry.

Also documented and discussed in the web at
https://github.com/ipld/go-ipld-prime/pull/144 .