GitLab

Both are now accessible.  Name is not always present.

Get rid of casts that are unnecessary.

Constructors for anonymous types are still upcoming;
all the current constructors dupe the name into the reference field.
Planning to add distinct methods on the Compiler for anon types.

As with parent commit: this is a checkpoint.  CI will not be passing.

This commit does not pass CI or even fully compile, and while I usually
try to avoid those, A) I need a checkpoint!, and B) I think this one is
interestingly illustrative, and I'll probably want to refer to this
diff and the one that will follow it in the future as part of
architecture design records (or even possibly experience reports about
golang syntax).

In this commit: we have three packages:

- schema: full of interfaces (and only interfaces)
- schema/compiler: creates values matching schema interfaces
- schema/dmt: contains codegen'd types that parse schema documents.

The dmt package feeds data to the compiler package, and the compiler
package emits values matching the schema interface.
This all works very nicely and avoids import cycles.

(Avoiding import cycles has been nontrivial, here, unfortunately.
The schema/schema2 package (which is still present in this commit,
but will be removed shortly -- I've scraped most of it over into
this new 'compiler' package already, just not a bunch of the validation
rules stuff, yet) was a dream of making this all work by just having
thin wrapper types around the dmt types.  This didn't fly...
because codegen'd nodes comply with `schema.TypedNode`, and complying
with `schema.TypedNode` means they have a function which references
`schema.Type`... and that means we really must depend on that
interface and the package it's in.  Ooof.)

The big downer with this state, and why things are currently
non-compiling at this checkpoint I've made here, is that we have to
replicate a *lot* of methods into single-use interfaces in the schema
package for this to work.  This belies the meaning of "interface".
The reason we'd do this -- the reason to split 'compiler' into its own
package -- is most because I wanted to keep all the constructor
mechanisms for schema values out of the direct path of the user's eye,
because most users shouldn't be using the compiler directly at all.

But... I'm shifting to thinking this attempt to segregate the compiler
details isn't worth it.  A whole separate package costs too much.
Most concretely, it would make it impossible to make the `schema.Type`
interface "closed" (e.g. by having an unexported method), and I think
at that point we would be straying quite far from desired semantics.

As discussed on the issue thread, ipld.Kind and schema.TypeKind are more
intuitive, closer to the spec wording, and just generally better in the
long run.

The changes are almost entirely automated via the commands below. Very
minor changes were needed in some of the generators, and then gofmt.

	sed -ri 's/\<Kind\(\)/TypeKind()/g' **/*.go
	git checkout fluent # since it uses reflect.Value.Kind

	sed -ri 's/\<Kind_/TypeKind_/g' **/*.go
	sed -i 's/\<Kind\>/TypeKind/g' **/*.go
	sed -i 's/ReprKind/Kind/g' **/*.go

Plus manually undoing a few renames, as per Eric's review.

Fixes #94.

So far, the generation itself runs, but the result will not yet compile.
I just want a checkpoint here.  (Most of this was written a few days ago already.)

This is full of mundane plonking away at adding stars and ampersands,
of which approximately none are interesting.

(I'm particularly frustrated by this because these are all placeholder
types, and we're getting *very* close to replacing them, as we get
closer and closer to self-hosting... at which point all of this bonking
about will be made totally irrelevant.  And yet to close the last mile,
these "small" fixes are surprisingly irritating.  Ah well.)

The bits that *are* interesting:

- the Spawn functions for type info now take strings rather than types
(so that they don't provoke a cycle problem for the user when
constructing the information to describe cyclic type info);

- all of the Type info structure hold a pointer to the TypeSystem, and
use that to look up reified Type info for related types, so that their
methods don't force the caller to do that themselves.

(The TypeSystem pointer was already there, amusingly; just never before
initialized, because it hadn't turned out to be load-bearing yet.)

It also would've been possible to just change all the methods on the
Type types to return TypeName rather than full Type info.  That would
avoid the need to use a TypeSystem pointer.  I didn't because:

Overall, this was done in such a way as to minimize the diff that
impacts within the templates.  This was a goal because updating
templates is a fair bit more work than other code due to the weak
compiler support.  And we'll end up reviewing and changing these
methods when we get to our goal of self-hosting generation of the
schema types from the schema-schema, so, it's not worth pushing around
diffs in that same area when they'd be sure to be churned under soon.

I can't quite claim tests passed on the *first* shot... but,
the first shot after mostly syntactical (rather than semantic) fixes?
Yeah, actually.  That was pretty fun.

Snuck in a bit of DRY'ing up.  The repetition in BeginMap methods
got to me, and was low hanging fruit, so I extract that from unions
and also backported it to structs.

Errors got some work in this commit, because it turns out I've
straightjacketed myself by not allowing "fmt.Errorf" due to imports.
There's a lot to do there, and I only tackled what was directly
critical to get this commit about unions across the finish line,
but there's a few remarks in comments about where to go next.

Some more comments about future work on the type info holder types
also appears; I'm starting to skid on those placeholders and their
issues more and more.  I really hope we can get to replace those
sooner than later.

And... also, yes, the idea of not having a "focus" state field in
assemblers really bit it, hard, as speculated in the previous
commit message.  I ended up using 'ca' in more switches than I
expected, simply because it's easier to use that than have the
conditonal templating branches that would be necessary to use the other
tagging mechanisms that do also have sufficient info.

One big fixme in the core interfaces for nodebuilders (wince):
the ValuePrototype method can error sometimes, and that hasn't been
accounted for.  Need to make a decision about what to do there.
It's not really an exercised path in practice, but it shouldn't
contain caltrops, regardless of how frequently used it is.
(This probably would've come up earlier, except there's a bunch of
stubs about ValuePrototype in other parts of codegen already; all of
them need backfill, but haven't yet made it to top of the todo heap.)

But despite all the fixmes accumulated, this does bring unions
to be a usable thing, and definitively proves out that the design
still works, even for what turns out to be one of the most complicated
parts of the schema system!  It's very, *very* exciting to add the
checkmarks to this part of the feature table -- it's one of the places
I most feared "unknown unknowns", now it's put to bed.  Woot!

Also make some of the placeholder construction functions.
(Not all of them.  Just the ones I intend to use first.)

There was some very unfinished placeholder stuff going on there.
(And ironically, it wasn't written in a very clear union-like way;
it gets a lot less icky when it's rewritten so that it's uniony.)

The appearance of the word "any" there has started to perturb me;
"any" is a concept that we also need to describe in schemas, and that
type has nothing to do with it.  It's more of a base mixin.

This diff is pretty fun.  It's our first alternative representation,
and it all works out nicely (no internal syntactical complications
encountered or any other unexpected unpleasantness).

It's also our first representation that involves potentially recursive
destructuring work over a scalar!  So, you can see the new 'construct'
method conventions appearing to handle that.  Works out neatly.
This kind of recursive destructuring happens all within the span of
processing one "token" so to speak, so naturally it can work without
any stateful machinery.

Some utility functions are added to the mixins package.
(It's sort of betraying in the name of the package, but, well,
it's an extremely expedient choice.  See comments about import
management.  tl;dr: consistency makes things easier.)

Tests for a recursive case of this will be coming soon, but requires
addressing some other design flaws with the AdjunctConfig maps.
(StructField is... not a good key.  It's not always hashable... such
as when the StructField.Type field is inhabited by TypeStruct.)
Easy enough to fix, just not going to cram it into this commit.

The way null and the 'fcb' are handled here differs from previous
generators: here, we *always* have an 'fcb', and just make a special
one for the root builder.  This is... well, it works.  It was an
attempt to simplify things and generate fewer method overrides, and
it did.  However, I'm not going to dwell on this too long, because...

The 'fcb' system is not working out well overall for other reasons:
it's causing costly allocations.  (Taking a function pointer from
a method requires at least two words: one for the function pointer
and one for the value to apply it on.  That means an allocation.)
So a serious rewrite of anything involing the 'fcb' strategy is needed.

And that is going to be a little tricky.  The 'fcb' idea was itself
a trying-slightly-too-hard-to-be-clever attempt to avoid an alternative
solution that involves generating additional types per distinct child
value slot (and I'm still unwilling to pursue that one -- it suggests
far too many types).  The next best idea I have now seems to involve a
lot of pointers into other assembler's state machines; this will surely
be a bit touchy.  But no worse than any of the rest of all this, I
suppose: it's *all* "a bit touchy".  Buckle up for fun diffs ahead.)

So, considering these 'fcb' notes, this feels a bit mixed...
"Two steps forward, one step back", so to speak.  Still: progress!
And it's very exciting that we got through several new categories of
behavior without hitting any other new roadbumps.

Using golang's plugin feature, we can... well, *do* this.

To date, testing codegen has involved running the "test" in the gen
package to get it to emit code; and then switching to the emitted
package and _manually_ running the tests there.

Now, running `go test` in the gen package is sufficient to do
*everything*: both the generation, and the result compiling,
and we can even write tests against the interfaces and run those,
all in one step.

There's also lots of stuff that becomes possible now that we can easily
generate multiple separate packages with various codegen outputs:

- Overall: everything is granular.  We can test selections of things,
  rather than needing to have everything fall into place at once.
- Generally more organized results.
- We can more easily inspect the size of generated code.
- We can more easily inspect the size of the compiled result of gen!
  (Okay, not really.  I'm seeing a '.so' file that's 4MB is coming out
  from 200sloc of "String".  I don't think that's exactly informative.
  Some constant factor is thoroughly obscuring the data of interest.
  Nice idea in theory though, and maybe we'll get there later.)
- We can diff the generated type for variations in adjunct config!
  (Probably not something that will end up tested, but neat to be able
  to do during dev.)

Doing this with go plugins seemed like the neatest way to do this.
It's certainly not the only way, though.  And in particular, I will
confess that this will probably make developing from a windows host
pretty painful: go plugins aren't supported on windows.  Mind,
this doesn't mean you can't *use* codegen or its results on windows.
It just means the tests won't work.  So, someone doing development
_on the codegen itself_ would have to wait for the CI server to run
the tests on their behalf.  Hopefully this is survivable.

(There's also a fun wee wiggle in that loading a plugin has the
requirement that it be built with the same "runtime".  The definition
of "runtime" here happens to include whether or not things have been
built in "race" mode.  So, the '-race' flag disappears from our CI
config file in this diff; otherwise, CI will get the (rather confusing)
error "plugin was built with a different version of package runtime".
This isn't really worrying to ditch, though.  I'm... not even sure why
the '-race' was in our CI script in the first place.  Must've arrived
via cargo cult; we don't _have_ any concurrency in this library.)

An alternative way to go about all this would be to have the tests for
gen invoke `go test` (rather than `go build` in plugin mode) on each of
the generated packages.  It strikes me as similar but worse.
We still have to invoke the go tools from inside the test;
we'd have *more* work to do to either copy tests into the gen'd package
or else generate calls back to the parent package for test functions
(which still have to be written against interfaces, so that they can
compile even when the gen isn't done, as it indeed isn't when you
freshly check out the repo -- exact same as with the plugin approach);
and in exchange for the extra work, we get markedly worse output
('go test' doesn't nest nicely, afaict), and we can't separate the
compiling of the generated code from the evaluation of tests on it,
and we'd have no possibility of passing information via closures should
we wish to develop table-driven tests where this would be useful.
tl;dr: Highest cost, uglier, and worse.

No matter which way we go about this, there *is* a speed trade-off.
Invoking the compiler per package adds at least a half second of time
for each package, in practice.  Worth it, though.

And on the off chance that this plugin approach does burn later,
and we do want to switch to child process 'go test' invocations...
the good news is: we shouldn't actually have to rewrite very much.
The everything-starts-from-NodeStyle-and-tests-against-Node work is
exactly the same for making the plugin approach work, and will
trivially pivot to working fine in for child 'go test' approaches,
should we find it necessary to do so in the future.  So!  With our
butts covered: a plugin'ing we shall go!

Some of the code here still needs cleanup; this is a proof of concept
checkpointing commit.  (The real thing probably won't have such
function names as "TestFancier".)  But, we do get to see here:
plugins work; more than one in the process works; and they work even
when the same type names are in the generated packages.  All good.

Some touches to ImplicitValue came along for the ride, but aren't
exercised yet.

Tests are getting unwieldy.  I think something must be done about
this before proceding much further or it's going to start resulting
in increasingly significant velocity loss.

Previously it was in the 'impl/typed' package, next to the
runtime-wrapper implementation of the interface.  This was strange.

Not only should those two things be separated just on principle,
this was also causing more import cycle problems down the road:
for example, the traversal package needs to consider the *interface*
for a schema-typed node in order to gracefully handle some features...
and if this also brings in a *concrete* dependency on the
runtime-wrapper implementation of typed nodes, not only is that
incorrect bloat, it becomes a show stopper because (currently, at
least) that implementation also in turn transitively imports the
ipldfree package for some of its scalars.  Ouchouch.

So.  Now the interface lives over in the 'schema' package, with all
the other interfaces for that feature set.  Where it probably always
should have been.

('typed.Maybe' also became known as 'schema.Maybe', which... does not
roll off the tongue as nicely.  But this is a minor concern and we
might reconsider the naming and appearance of that thing later anyway.)

Signed-off-by: Eric Myhre <hash@exultant.us>

The schema.Type objects don't have a `Kind() ReprKind` method.  That
was just a bad idea.  I didn't try to fix it; I just nuked it.

The new schema.Type.Kind() method docs include a description of why
it's incorrect to wish for a either a single or a statically known
ReprKind for a type.

(Context mention: the previous commit was also a wind-up for this.
And technically, I'm still winding back up the mental stack here:
this had to get straightened out because I wanted to start poking
codegen for structs; and then I had beef with the TypeStruct.tupleStyle
field being a bool; and then fixing that pointed out that the current
Type.ReprKind methods were kinda insane... yaks!)

- `typed.Node` -> now lives in the `impl/typed` package, more like
other nodes.

- Most of the other essential parts of reasoning about types moved
to `schema` package.  ("typed.Type" seemed like a nasty stutter.)

- `typed.Universe` renamed to `schema.TypeSystem`.

- Current `Validate` method moved to `schema` package, but see new
comments about potential future homes of that code, and its
aspirational relationship to `typed.Node`.

Conspicuously *not* yet refactored or moved in this comment:

- The `typed/declaration` package -- though it will shortly be scrapped
and later reappear as `schema/ast`.  The dream is that it would be
neatest of all if we could generate it by codegen; but we'll see.
(This would seem to imply we'll have to make sufficient exported
methods for creating the `schema.Type` values... while we also want
to make those immutable.  We'll... see.)

- The `typed/gen` package is also untouched in this commit, but
should similarly move shortly.  The codegen really ought to be built
against the `schema.Type` reified interfaces.

Overall, this drops the sheer nesting depths of packages a fair bit,
which seems likely to be a good smell.

Most of the meat is implemented, with a few important exceptions.

For many of the more simple kinds of type, it's pretty cookiecutter:
e.g., for bools and strings and the other simple scalars, it's just
copying over the descriptions to the immutable objects in the
typed/system package, and it's done.

For maps and arrays and other recursive types, a little more code
is involved: we do a two-pass approach to make sure we get all the
named type declarations enumerated first, and then go back to review
the recursive kinds again, and now check and reify their recursive
references.  That's pretty straightforward as well.

The one problem here is... we didn't account for inline anonymous
type definitions, e.g. the `struct { field {Foo:[Bar]} }` syntax.
That requires some significant fixing in the typed/declaration
structs; they don't have the right recursivity at all right now.

I started punting out to todo comments rather than properly finishing
the construction of the struct and union types after making that
realization about anonymous inline type definitions.  I'd like to
hammer out the correct recursivity in the declarations types before
coming back and doing a final finishing pass on this.

And one more side-note: I turned TypeName into an alias between the two
packages because it reduces syntactic irritation and casting.

Up next: well, we did all this first pass work on typedeclarations, but
something's been under-regarded: a core design goal is to make sure
these schema declarations are self-describing and serialize clearly
in the IPLD Data Model formats.  Let's spend a bit of time on that.
(What I *really* want is to have *codegen* make the whole suite of
the typed/declaration package contents straight from a schema!  But
we've got a bit of a bootstrapping cycle there, of course, heh heh.)
Signed-off-by: Eric Myhre <hash@exultant.us>

ISTM that since we're talking about raw data more so than "objects" in
any kind of Alan-Kay/sending-messages/OOP sort of way, calling it
"struct" is more accurate communication.

(It's also a lot less likely to be confusing when mentioned in the same
context as javascript, since "object" and "map" tend to be collisiony
terms there; whereas "struct" just doensn't really have any prior
associations in that community as far as I know.)
Signed-off-by: Eric Myhre <hash@exultant.us>

While fixing this it occurs to me that we could in theory also support
more than one name alias that will be mapped onto the same concrete type.
...
But Let's Not with that.  It would increase the code complexity and more
importantly, *it would be lossy to do so*.  So let's not open that can
of worms: straying into lossiness territory is far worse than whatever
other hypothetical problems one might imagine trying to solve.
Signed-off-by: Eric Myhre <hash@exultant.us>

...and all fields are correspondingly unexported.

We're going full immutability on almost all this stuff.  I simply can't
imagine handling and maintaining this any other way.

The typesystem.Type interface also got a few more standard methods.
I'm not sure if having a Universe backpointer is going to be useful
very often or not, but it seems reasonable enough to do; we'll
definitely *have* that information during consistency checking and
reification time, so it should turn out easy to keep it.

The validate_test.go code is now looking a bit janky.  We need
initializers for the typesystem.Type* values.  Those will be the
typedeclaration.* package... which we'll write next.

Note that I skipped any accessor methods on `Type*` objects which are
for representation details.  These might get filled in later, but I'm
gonna see how far I can get without them.  (So far, there's some
uses of the representation details in the validate method, but we've
located that in the typesystem package anyway, so that's... fine.)
Signed-off-by: Eric Myhre <hash@exultant.us>

See previous commit for more discussion of the child packages to come,
or the schema.md file in the docs dir, which describes similar.

Several large todos in comments.

Quite importantly: typed.Node is now an **interface**, not a concrete
struct itself.  The godoc comment should explain this nicely; long
story short, this is going to be relevant when we get to codegen and
other advanced form of native integration.
Signed-off-by: Eric Myhre <hash@exultant.us>

And ReprKind moves from the typed package to the ipld main package.
It's hard to get too much done without the standardization of ReprKind.

Between the Kind() and Keys() methods, it should now be possible to
perform traversals of unknown nodes.

This diff just worries about implementing all the Kind() methods.
Keys() has some additional questions to handle (namely, map ordering?).
Signed-off-by: Eric Myhre <hash@exultant.us>

Having a Name() interface method might also have been useful, but at
the moment, it's too annoying.  Maybe I'll come back to this and add a
ton of constructors for each kind of type and make all their fields
private scope, which would solve the name collision... maybe.  Later.
(This is all expected to be implementation-internal stuff in the long
run rather than anything user-facing API, so it's up for debate how
much polishing it's actually worth.  Unless that changes!)

The ReprKind method is *usually* pretty predestined based on the kind
of type in the first place, but a few cases are interesting.

... *Especially* kinded unions.  I'm a little alarmed at the break of
pattern, there.  Hopefully that doesn't manifest too much complexity
down the road.  But if it does... eh, well... nature of the beast.
Kinded unions are definitely a useful feature.
Signed-off-by: Eric Myhre <hash@exultant.us>

There's... not really much to it.
Signed-off-by: Eric Myhre <hash@exultant.us>

Giving a type to each, um, kind of type.

Syntax draft changed.  Bigger examples elsewhere.  May need to start
writing up more, but might not belong in this repo, either (it's not
a detail of the golang implementation alone).  Anyway, it's only for
eyecandy at present: I'm not going to start work on a parser for the
DSL any time soon; it's just something to keep in mind so it can be
evaluated against simplicity heuristics.

Struct fields are in a slice now.

Waffling on whether to call that kind "struct" vs "object".  Neither is
very satisfying.  "Struct" is suggestive of something with concrete
memory layout considerings; this is not that.  "Object" is suggestive
of something with methods and behaviors attached; this is not that.

Introduce an enum for different kinds of union.  Some of the terms here
are developed from https://github.com/ipfs/unixfs-v2/issues/20 ; the
term "keyed" is from https://github.com/polydawn/refmt/pull/30 ; and
the style "kinded" is a concept I haven't written up much about before,
but the TypeUnion.ValuesKinded field type should self-explain it.

Flip terminology from "required" to "optional".  Struct fields that are
mentioned in the schema default to being required; why else would you
have mentioned them?  Previous draft of DSL which used question marks
and exclamations points to denote these things dropped; the new hope is
that *neither* nullable nor optional will be used with enough frequency
that the sigil-heavy shorthand will turn out to be desirable.
Signed-off-by: Eric Myhre <hash@exultant.us>