GitLab

An "immediate" rather than generative function is also available;
the docs contain caveats.

At the moment, these distinctions are a bit forced, but we want to be
ready for when we start getting truly huge maps where the generative
usage actually *matters* for either memory reasons, latency reasons,
or both.

Separated Length rather than trying to pull double-duty the Keys
method; the previous combination was just utterly silly.

The implementations in the bind package are stubs; that package is
going to take a lot of work, and so the focus is going to be entirely
on keeping the 'free' package viable; and we'll revisit bind and such
when there's more dev time budget.
Signed-off-by: Eric Myhre <hash@exultant.us>

Signed-off-by: Eric Myhre <hash@exultant.us>

Continues the work started in 2263295e, with the purpose of fixing
undesired three-valued logic by disambiguating 'optional' from
'default'.  Some comments in the schema now address this in detail,
including description of the membership cardinality.

Some incongruities in the JSON have been fixed;
and a significant new set of definitions for struct-represented-as-map
configuration details is introduced.

I gave some brief consideration to eliding StructRepresentation_Map
and instead simply using a typedef'd map, and it would be possible
since it only has one field... however, I'm opting to keep it because
I'm willing to bet we'll come up with more properties someday which
are for the whole struct rather than just per field, and thus it's
unlikely that StructRepresentation_Map will stay single-field forever.

The 'default' value itself in the new
StructRepresentation_Map_FieldDetails type is interesting: it's the
first time we've had an excuse to use an "Any" cop-out.
And even more interestingly, it's almost certainly invalid: as
commented inline in the diff, that field would probably be even better
represented as a kinded union which makes it clear that only the scalar
primitives are welcome here.
We'll almost certainly introduce that union in a future commit, but
I'm gonna let that thought marinate for a while first.
There are related interesting questions about how we want to handle
the DSL parsing in this area, and those might be good to think about
more deeply before finalizing this part of the schema.
Signed-off-by: Eric Myhre <hash@exultant.us>

Signed-off-by: Eric Myhre <hash@exultant.us>

I'm using templates for this rather than AST libraries because after
some review of existing systems, it seems that this is a popular
choice; and I'm flat out going to assume those people have known
what they were doing and made their choice rationally.  Protobufs,
graphql codegen libraries, the ugorji serialization libraries -- almost
every codegen I've ever come across uses templating.  So, we will too.

And so far, the results seem promising.

The tests here will be shoddy for some time.  I'm outputting code to
a directory and periodically eyeballing it and making sure the compiler
likes it.  My aim is to get some fair fraction of codegen bootstrapped
this way; and after we start codegen'ing the typedeclaration package
itself, *then* we'll start solidifying things with regression tests.

There's a lot of discovery process to go, here.  See for example the
comments about ranging versus iterators (namely, that it's not
possible, which... turns out is going to feed back into our design
all the way back out at the layer of the ipld.Node interface itself).
Signed-off-by: Eric Myhre <hash@exultant.us>

Removed the 'name' field from all of them.  Doesn't match schema.
While it's true that we'll want a name property on the reflective
typesystem.Type objects, that's... well, that's over there.

I'm starting to make fairly wild zigzags through here; with reason:
I'm trying to get as quickly as possible to a bootstrap of the essenses
of codegen.  The hope is to get there fast, and power it with a few
hardcoded values; then get that to churn out a replacement for the
hand-written declarations package types; and then teach the codegen
to write more and more of the e.g. ipld.Node interfaces and then
the serialization bridges and then the NodeBuilder stuff, etc.
The core coverage for getting the codegen to go, though, I think it's
actually sufficiently smaller than all those parts that maybe this
will be reasonable to pull off this way.  Time will tell.  (Shortly!)

Dropping the typed/system/construct.go file temporarily, as the changes
to the other typedeclaration files has broken it, and maintaining it
through the shaking seems like it will generate inordinately more
work than it's worth.  It'll be revived soon; it's mostly on the right
track, it's just not on the critical path atm.
Signed-off-by: Eric Myhre <hash@exultant.us>

This change to the schema-schema in DSL form makes it accurately match
the schema-schema in JSON form, *AND* fixes the undesired three-valued
logic at the same time.  Wunderbar.
Signed-off-by: Eric Myhre <hash@exultant.us>

And fix a few bugs detected in the DSL form (one union typo'd into an
incorrect representation; and one just outright missing its
representation declaration), but fortunately nothing too serious.
Signed-off-by: Eric Myhre <hash@exultant.us>

The JSON representation of the schema shortens up noticeably if the
properties for "nullable" and "optional" are themselves optional.

Given that the hope is for the in-practice use of these properties to
be the exceptional case rather than the norm, it's nicer to speak
less of them.

This edges around an interesting question: we don't actually *want*
three-valued logic here... but that's precisely what "optional Bool"
gives you: it may have the value "true", "false", or "undefined"!
What we *mean* here is of course "true" or "undefined".  It would be
desirable to have a way to encode that.  Perhaps we can find a way.
One idea is that this is an issue in the representation of bool;
and thus perhaps we should configure it in the struct's representation
block rather than marking it optional in the struct's definition block.
Let's give this more thought in the future.
Signed-off-by: Eric Myhre <hash@exultant.us>

(An alternative proposal, with the same example DSL but different
choices in schema representation is in a sibling commit, and we'll
merge and pick one or the other to continue with shortly.)

This take on TypeTerm keeps things flatter and has fewer schema types:
the optional and nullable properties are those of the containers
in the recursive type (i.e., the map itself, list itself, or the fields
in the case of structs).

As with the alternative proposal, this seems to work and capture all
of the concepts that we want:

- the inline definitions are possible;
- they recurse correctly
- nullability can be inserted in all the correct places;
- you *can't* have a "nullable nullable" nesting;
- and bonus, the simple TypeName case is still a string repr.

So, since all the same key properties exist in both approaches,
in some ways, this is six-of-one/half-a-dozen-of-the-other.
Both mechanisms of encoding clearly represent the same things.
It's just question of which is nicer to work with, which seems more
self-documenting to a human reader, and other intangibles of that sort.

In this approach, we have a less rightward-leaning tree when the schema
is represented in JSON form; by comparison, the alternative which had
"nullable" represented as a union experiences deeper nested objects.
Reducing depth-of-tree is hardly a proof of simplicity, but it's usually
a good heuristic, so we can count that as a win here.

Another comparison we can make is that the number of types used in this
diff is 2; the number used in the other approach was 4.  We do this
at the trade of more fields in existing types, of course, so it's less
than perfectly clear cut as a win for simplicity... but I tend towards
thinking it is.  Especially certainly clearer is that we *don't* have
a union type which immediately nests another different representation
of union type in this approach!

So, while it's always hard to make absolutionist statements about
"simplicity", indeed, this seems simpler via several observations.

In terms of what other people are doing...
One interesting example from another endevour I recently noticed -- the
Atom project emits a JSON file describing their API:
https://github.com/atom/atom/releases/download/v1.34.0/atom-api.json
In this JSON file, they've taken a tack similar to the one emboded in
this commit: the "isOptional" property is inline with the field defn's.
So, we can count Atom's choice as an effective a +1 vote for that.
(It's perhaps interesting they also have *names* in their field obj...
however, the fields here are in an array, which is necessary since
they're describing positional arguments.  So, that's entirely
self-consistent, but also different constraints than we have, and thus
while interesting, not quite comparable to our choices on that matter.)

Note that the JSON-formatted schema-schema file has not been updated yet.
We'll update and finish that after merging and selecting a proposal here.
Signed-off-by: Eric Myhre <hash@exultant.us>

This seems to work and capture all of the concepts that we want:

- the inline definitions are possible;
- they recurse correctly
- nullability can be inserted in all the correct places;
- you *can't* have a "nullable nullable" nesting;
- and bonus, the simple TypeName case is still a string repr.

The downsides are:

- wow, this *looks* complicated on the schema side;
- in part because we have a union-union nest, and conceptually we'd
   probably simply prefer to unify those, but it's a representational
   problem (can't clearly combine a keyed and a kinded);
- the json repr looks to get deep fast.

I wonder if we can do better than this.

And also, we shifted the flagging of optional fields to
StructRepresentation_Map, then eliding a struct for field properties
entirely.  This seemed potentially sensible given that we suppose a
struct can be represented as a list, in which case optionality of
fields won't work the same.  However, on further thought, I'm also
extremely dubious of this idea: whether or not a field can be traversed
and then yield "undefined" is defined by whether it's optional; and
I'm really suspecting it's going to create a lot of weirdness to make
that feature conditional upon the StructRepresentation union.
If anything, we should simply make invalid combinations of optionality
and representation of that struct, well, erm, invalid.
Signed-off-by: Eric Myhre <hash@exultant.us>

Even though we having concretely specified TypeTerm yet, it's a pretty
fair bet that map and array value types will both use it.
Signed-off-by: Eric Myhre <hash@exultant.us>

StructRepresentation also offers at least two options; and in the
future, we might anticipate adding more options as well (suppose for
example some kind of struct-to-string conversion systems; of which
their could be many, but imagine for example one like matching a
string like the "options" in your /etc/fstab: simple comma separated
keys and values should be definable enough, no?).

And now we have to wrangle the TypeTerm recursivity to completeness:
how shall we concisely and correctly represent the declaration of
inline declarations of map and array kinded types?  I've left this
a big honking TODO for this commit; it's touchy.
Signed-off-by: Eric Myhre <hash@exultant.us>

Not having a "set" definition is kind of irritating.

I'm wondering if we should propose to make a set kind; and simply
transform these to a map with null values as necessary.  Possibly it
could even be argued that this could make sense at the Data Model
level?  I'm not sure.  Something that should be discussed, though.
Signed-off-by: Eric Myhre <hash@exultant.us>

Signed-off-by: Eric Myhre <hash@exultant.us>

To prove that we can.  And now that we've got unions defined, we can
actually describe most of the core concepts of "Type" itself, etc.

Fixed one bug in the DSL-style schema; the union discriminator keyword
for what kind of type a type is should be... "kind".  Whew.
Signed-off-by: Eric Myhre <hash@exultant.us>

It is indeed redundant with the maps that must necessarily exist in
each of the representation union members.
Signed-off-by: Eric Myhre <hash@exultant.us>

This is actually a fairly big improvement on what we'd written for the
internal implementation in Go.  The representation *should* be a union,
in general.  We'll do something similar for structs, and everything
that has multiple options for representation strategy.

One FIXME bugaboo: it's now absolutely unclear what to put as keys in
the members map, as although there is a key, which one to use depends
on the representation kind.  It would feel odd to not have the
membership be clear before reading through to the representation info,
but I guess that is defacto where that information lives, so...
Signed-off-by: Eric Myhre <hash@exultant.us>

Signed-off-by: Eric Myhre <hash@exultant.us>

Added some enums that should be useful, and documented the bejezus out
of them.

Seems to be going well so far.

Documention for TypeUnion.

All the other interesting Type types are still todos, though; and we
thus far haven't hoisted ourselves back up to the point where typeterm
recursiveness needs to be fleshed out, which is one of the things we
already know is going to be Fun.

Fixed what's probably a typo with an extra equals sign.  I'm not sure
why my brain keeps wanting to do that for unions.

And one interesting note: though I'm keeping TypeKind in the commit,
see the todo comment about its potential irrelevance?  It's amusing
that we've just parallelly reproduced that; the same misstep already
happened once before, just in the golang code: TypeKind there was also
not needed, because the Type union members already represented it.
Signed-off-by: Eric Myhre <hash@exultant.us>

Everything here so far is probably good; need to continue specifying
all of the individual type structure next.
Signed-off-by: Eric Myhre <hash@exultant.us>

Most of the meat is implemented, with a few important exceptions.

For many of the more simple kinds of type, it's pretty cookiecutter:
e.g., for bools and strings and the other simple scalars, it's just
copying over the descriptions to the immutable objects in the
typed/system package, and it's done.

For maps and arrays and other recursive types, a little more code
is involved: we do a two-pass approach to make sure we get all the
named type declarations enumerated first, and then go back to review
the recursive kinds again, and now check and reify their recursive
references.  That's pretty straightforward as well.

The one problem here is... we didn't account for inline anonymous
type definitions, e.g. the `struct { field {Foo:[Bar]} }` syntax.
That requires some significant fixing in the typed/declaration
structs; they don't have the right recursivity at all right now.

I started punting out to todo comments rather than properly finishing
the construction of the struct and union types after making that
realization about anonymous inline type definitions.  I'd like to
hammer out the correct recursivity in the declarations types before
coming back and doing a final finishing pass on this.

And one more side-note: I turned TypeName into an alias between the two
packages because it reduces syntactic irritation and casting.

Up next: well, we did all this first pass work on typedeclarations, but
something's been under-regarded: a core design goal is to make sure
these schema declarations are self-describing and serialize clearly
in the IPLD Data Model formats.  Let's spend a bit of time on that.
(What I *really* want is to have *codegen* make the whole suite of
the typed/declaration package contents straight from a schema!  But
we've got a bit of a bootstrapping cycle there, of course, heh heh.)
Signed-off-by: Eric Myhre <hash@exultant.us>

The typed/system package calles these members, so the typed/declaration
package should too.
Signed-off-by: Eric Myhre <hash@exultant.us>

There are several "REVIEW" comments here, so this will almost certainly
see another round of iteration.

There are, of course, clear enough ways to unify this and Transform
into one general function.  The tricky questions are entirely around
what's the best way to make the simple cases syntactically simple...
without trading it for either easily-misused APIs or unreasonable
syntactical complexity for the more complex compositional cases.

(It should also be noted I'm doing morning cleanup commits of diffs
that were produced in the past when *probably* having been up too late.
Please be gentle in your thoughts, future reader.)
Signed-off-by: Eric Myhre <hash@exultant.us>

Upon a good night's sleep and reexamining the situation, it's extremely
clear that this function is *not* a fluent thing at all.  (Duh; look at
the Node types.)

A non-fluent Transform method has tons of virtue.  In many cases, we
might not actually need the fluent.Node helper features at all when
using Transform: since all of the intermediate error handling to
traverse to the reachedNode is already handled, that's half the battle;
and whether or we need to fight the ease-of-use battle on the interior
function is... *it depends*: if we use (codegen'd) native types, those
already address a lot of the error handling; fluent.Node can also be
used on the inside, but it's simply not needed in all cases.

We'll come back to the subject of writing the fluent equivalent later;
some example usage development is needed first so we know how to make
something that's actually helpful.
It will, of course, mostly be a wrapper; the interesting question is
whether a fluent.Transform function should use a fluent.Node either,
since this will make it harder to compose (unless we make fluent.Node
still embed ipld.Node?)... ahem.  As I said.  We'll come back to that.

One more quick comment about this relocation: I'm a little tepid about
putting the Transform method in the main package, because that really
puts it on-screen quite in the forefront.  If it's just Transform and
Traverse, maybe this is fine; but we end up implementing a large suite
of optics methods, this will become cluttery.  So, potentially this
will move to another sub-package again if that becomes an issue. (But
it still wouldn't be the fluent package.)
Signed-off-by: Eric Myhre <hash@exultant.us>

These tests are all Path.Traverse tests, not general Traversal
interface tests or specs.

And while having the poor Traversal interface all alone in its own file
seems a bit lonely, ... well, maybe we'll give it some company in a
second.
Signed-off-by: Eric Myhre <hash@exultant.us>

It's really a mouthful.  I think this is a little better, at least.
Signed-off-by: Eric Myhre <hash@exultant.us>

The docs in the diff body do most of the explaining work here.

I'm not sure if these should be in the fluent package or not.
Probably it will be wiser to have a non-fluent variant first, and wrap
those in the fluent package.  I wrote them here first, though, so,
here they stay for this commit.

I'm checkpointing this here, but there's at least one major issue I've
already realized: with these interfaces, we can't keep correct paths
when we do *nested* Transform calls.  Perhaps that's survivable; but I
find it very unattractive.  So, we'll do some additional work to hammer
that back out -- the tricky part will just be doing it without
increasing the amount of complexity that's user-facing.

Also -- yes, I know the parent commit says "we'll get rid of
MutableNode" next -- but it's basically this Transform stuff that's
been the driving force behind most of the other traversal and builder
work going on today, so...  We'll nudge both along in tandem for a bit.
Signed-off-by: Eric Myhre <hash@exultant.us>

Specifically in regards to typed nodes, which are the main consumers of
this that I imagine right now (though we may find others in the
future).

Typed nodes which are implemented by *native* concrete types (e.g.
presumably by codegen -- though we haven't implemented any of this yet)
will *especially* hit this detail: they'll return builders that produce
new instances of their own concrete native type!

And all of the build methods that actually produce a new Node now also
have the option to return an error.  This follows from the docs;
typed nodes actually can and will reject some attempts at invalidly-
typed creation!

We'll probably want fluent wrappers of NodeBuilder as well, as a result
of these error returns.  I'll defer that until it's forced.
Lazy/just-in-time evaluation of the programmer :P
Signed-off-by: Eric Myhre <hash@exultant.us>

This will replace, deprecate, annihilate, obviate, oblivate, and
generally nuke the MutableNode interface.  MutableNode was a bad, bad
idea and made many things terribly complicated.  Immutable interfaces
are shaping up much, much better and I'm excited to move NodeBuidler
forward.

I tried for some time to figure out a useful way to do a builder
pattern in the fluent package, which would work *upon* the MutableNode
interface... but all attempts to do that ended poorly with code that
was both messy internally and complex and error prone for callers.
I've given up all such attempts.

It also seems to be true that we should have NodeBuilder *instances*
which may be per Node.  This wasn't obvious at first, but I think it'll
let us carry a lot of important information in a low-fuss way, and it
solves a *lot* of problems that cropped up when trying to implement
larger scale generic transformation methods on Node graphs (namely, the
frequent need to be able to create a new Node in the middle of a
process; and handing down a factory method to each of those points was
horrifically verbose and caused complexity that was not essential).
(Unfortunately, I can't really point at examples of *how* bad this got,
because I've never committed any of those research spikes; fortunately,
no one else will now ever have to *see* that mess, since we've got
better solutions figured out.)

MutableNode will go away in the next handful of commits, but I wanted
to land the new one first in a smaller diff.  Removing MutableNode
may be a fair amount of messy work, because of how it's used in nearly
all of our test fixture setup to date!  Uffdah.

(Node will get a GetBuilder method (or something of similar intent but
better name) at the same time we get rid of MutableNode; I'm eliding
it from this commit for the same reasons about diff size avoidance.)
Signed-off-by: Eric Myhre <hash@exultant.us>

Signed-off-by: Eric Myhre <hash@exultant.us>

The Traversal interface is a tad verbose to implement yourself;
the idea is that you *usually* won't, but it's a powertool that's there
if you really need it.

Some other optics-like functions will be coming up soon.
Signed-off-by: Eric Myhre <hash@exultant.us>

With a panicky method.  This clearly needs to be done... eventually.
I'm currently just embarassed I can't `go test ./...` and that
needs to be fixed *now*.
Signed-off-by: Eric Myhre <hash@exultant.us>

It should return an error explicitly.

A pair of "(nil, nil)" reponses would also be an unambiguous option for
representing an absent member, but seems more likely to create pitfalls
than save a significant amount of code for callers.
Signed-off-by: Eric Myhre <hash@exultant.us>

ISTM that since we're talking about raw data more so than "objects" in
any kind of Alan-Kay/sending-messages/OOP sort of way, calling it
"struct" is more accurate communication.

(It's also a lot less likely to be confusing when mentioned in the same
context as javascript, since "object" and "map" tend to be collisiony
terms there; whereas "struct" just doensn't really have any prior
associations in that community as far as I know.)
Signed-off-by: Eric Myhre <hash@exultant.us>

All of these struct definitions are horrifying similar in construction
to the same-named definitions in the typesystem package: simply ever
instance of a Type pointer is instead a TypeName string.  Despite being
unattractive, this is correct: since typesystem can have pointers and
cycles, and typedeclaration can't, this is what we end up with.
Signed-off-by: Eric Myhre <hash@exultant.us>

While fixing this it occurs to me that we could in theory also support
more than one name alias that will be mapped onto the same concrete type.
...
But Let's Not with that.  It would increase the code complexity and more
importantly, *it would be lossy to do so*.  So let's not open that can
of worms: straying into lossiness territory is far worse than whatever
other hypothetical problems one might imagine trying to solve.
Signed-off-by: Eric Myhre <hash@exultant.us>

Signed-off-by: Eric Myhre <hash@exultant.us>