GitLab

Fix interaction between ExploreRecursive and ExploreUnion

More complete selector traversal

Builder for selectors

Feat/recursive selectors

Fixes a bug where ExploreRecursive would not properly replace ExploreRecursiveEdge (and cause a
panic when traversed) if ExploreUnion was part of the recursion tree.

Properly record last block traversed

refactoring functions in traverseInformatively to simply and optimize logic and also minimize inner
loop function calls

Improve the traverse function so that it only explores relevant fields for low cardinality selectors
and traverses high cardinality selectors for lists correctly

Add link loading to traverse informatively so that you can traverse across links

Decoding of nodes that are just simple values was erroring for both CBOR & JSON. This fixes the
issue as well as fixing it in the general unmarshal.

Extract the selector buidler to a seperate package

Adds a builder class to quickly assemble selectors nodes, using a DSL that enforces the underlying
schema. This allows quick assembly of selector nodes. This will likely be removed once IPLD Schema
code gen is finished, but for the time being, it provides a quick way to ensure mostly typesafe
selector nodes.

Rename SelectorContext to ParsedParent (more accurate) and put in a ParseContext state object which
is tracked w/o var args

Support validating recursive selectors to match ExploreRecursive with ExploreRecursiveEdge

An initial implementation of recursive selectors that offers a way to traverse them. Not covered is
validating ExploreRecursiveEdge being under an ExploreRecursive selector

Mind, I don't know if all the details of this have been worked out.
In particular, where to get the LinkStorer and other fun stuff passed
through this interface might be a fun dance (if, indeed, such a thing
ends up needed -- this hasn't been written yet!).  We'll see.
One thing is clear: we want this contract here for all the same
already-known reasons: a traversal.Transform over a structure using
an ADL should "just work".
Signed-off-by: Eric Myhre <hash@exultant.us>

Prompted in part by
https://github.com/ipld/go-ipld-prime/pull/21#issuecomment-518115245 .

Turns out some of the stuff our earliest Node implementations do are
actually pretty unusual.  Many other features have more strictures.
Therefore, it's important to specify the behavior such that the
features that have to operate within more restrictions can still
comply, and it's clear that anything else shouldn't be expected!

(Perhaps we should just... make the ipldfree.Node implementations not
do the flexible things they do right now.  That package is starting
to look in need of some renovation anyway.  We'll see.)
Signed-off-by: Eric Myhre <hash@exultant.us>

Schema and codegen fun

fix(encoding): match go-ipld-cbor cid encoding

go-ipld-cbor encodes a 0 byte ahead of cid bytes for multibase support - to maintain compatibility
of nodes encoded to CBOR, add this feature

This one was particularly aimed at making sure I hit all the hardest
situations as trying to write the generators for NodeBuilders.

It certainly did so!  I found enough conceptual/design work to do that
I'm actually going to park this here for a while, probably land this
branch back to master before it gets any longer, and then approach
things from some other angles for a while.

In particular: We're going to need several *different* NodeBuilders
for schema-typed nodes.  The astute reader probably already noticed
this, as it's implied by the `typed.Node{}.Representation() Node`
method: that adds a second Node instance, and that means a second
NodeBuilder.  What's a little more fun yet is there might be more than
two: for example, if we want the serial mode to support variation in
field order for structs, that's another branch we need to account for
somewhere... and, ordered vs unordered modes might need different
structures of memory to do their job efficiently, so that means wholly
different implementations of the interface.  Whee.

So.  I'm thinking I might switch tack to knocking around in the
(non-codegen) typed Node package for a while and making sure nothing
looks wildly amiss over there while exploring these same features
Then, when coming back here: looks like we're due for that breakdown
of `EmitNodeBuilder{Detail...}` methods: and perhaps that's
actually going to belong on a new type rather than glommed on flatly
to `typeGenerator`... since there may be a n:1 relationship there!

A bit fun; wish there was a way to compress that block of ifdefs about
optional vs nullable stuff, but so far nothing clearer has emerged.

Added ErrIteratorOverread while at it.  Should refactoring other node
types to use it too, I suppose.  But perhaps we'll just accumlate those
todos for a while, since there's already one other error refactor with
a chain of blocks in front of it.

The comment in the ipld.Node type about IsUndefined has been there
quite a while; today we're finally using it.

With this set of changes, the generated code for structs now compiles
successfully again: the Undef and Null thunks are both used there.

The latter isn't used yet; that's not the main goal of this branch (and
I think I'd rather do the refactor to use ipld.ErrNotExists after
moving the PathSegment types from the selector package into the root...
which in turn I'm not going to do while some of the other current work
on Selectors is in progress in other branches.  So!  Latur.); I just
wanted it around for documentation and clarity of intent.

This also adds a couple of other dummy references to the 'typed'
package.  I know keeping a set of packages-used-in-this-file is a
common step in the development of codegen systems targetting golang,
but I'm going to try to avoid it until the need is really forced.

Generates the struct itself (including the variations of pointers
and/or extra fields necessary),

There are several major TODOs outstanding; and the generated code
doesn't *quite* compile due to several missing references, namely:

- ipld.ErrNoSuchField
- ipld.Nil
- ipld.Undefined

The other issues remaining:

- The reason ErrNoSuchField is missing is because it needs to be
decided where an error like that should go.  Is it meaningfully
distinct from a general "key absent" error, like a map can return?
If it is indeed a distinct thing, does it go in the schema package,
or will that just make things annoying until the end of time since
every occurence-site-sibling error is in the ipld main package?

- We need to think about typed nil / typed undefined.  I suspect they
are an idea to be avoided (but doing so means using the NodeBuilder()
method on the nil or undefined values from a struct will *not* "DTRT",
which may be problematic; this can in turn be avoided by "knowing"
you're working on a typed node, but, erm.  It turns into a question of
which of these things is less annoying).

- We'll need to backtrack up to the main ipld.Node interface and add
that 'IsUndefined() bool' method now!  (Or, take a radical approach of
using a magic const for that.  But... no, that doesn't sound fun.)

- Haven't even touched the NodeBuilder yet.  (And the typeGenerator
interface needs to have those bits broken down, as well, so that we can
do good stuff with error path codesharing as with funcs for Node.)

- Haven't filled in MapIterator.  But that's probably just easy chug.

Lots to do, in short.  I just need a checkpoint.

Note how we've had this on the clipboard for a while already, heh --
the info_test.go file and TestUnderstandingStructMemoryLayout have been
aimed at this; notice how any extra bool fields are generated at the
bottom of the emitted struct.  Many yaks have been shaved in the
meanwhile, and yet look at how many we have to go.  Isn't this fun?
Signed-off-by: Eric Myhre <hash@exultant.us>

Signed-off-by: Eric Myhre <hash@exultant.us>

Fields in these structs are now named.  Embedding is reserved for
things that are very explicitly supposed to carry their methods along.

At some earlier point, I embedded the Type field in some of the
generate structs because it saved me a few keystrokes in refactoring
template strings.  This was not important.

Today I learned: if you have embeddings which cause a name to be
shadowed -- as the generateKindedRejections_String.Type field
could shadow the generateKindString.Type field, for example -- if those
resolve "neutrally" (I don't know if there's a better technical term
for this), then... in that case, the templating system will be like
"yeah, fine, whatev", and it works.  If you give one of those two
fields a *separate type*, the templating system will now reject it as
if no field of that name can be found.

I encountered this while working on the generator for structs (which I
*swear* I will commit any second now, when I stop finding prerequisite
yaks to shave -- there have been many) and trying to use a more
specialized TypeStruct instead of the Type interface.  Whee.

I get the logic of this.  The error message could be better.

Anyway: even though it technically didn't affect generateString (yet),
making the change here for consistency with what's to come
(including specializing the field even though there's currently no
actual need for it).

This isn't a net negative size diff yet, but it certainly will have
that effect momentarily when we add a generator for another kind.
More importantly, it reduces the set of functions in the real
generator file to *just* the relevant ones.

We'll almost certainly extend this to cover the NodeBuilder half of
things as well, later; I just haven't gotten there yet.

I wanted this for use in codegen work: in particular, as part of
generating error messages.

Note the comment on the case for unions.  I still go back and forth on
this topic; see some recent previous commit messages on this branch.
(I think I'm slowing coming around to "unions act like maps", and it's
just that most library code chooses to jump over them transparently,
because that's the least-odd path for resolving NodeBuilder questions;
but it remains the case that none of this is settled for sure yet.)

Update to selector spec, minus recursive selectors

update comment to clearly indicate return value correctly

update the structure for ExploreIndex & ExploreRange to be slightly more clear and optimized

Adds unit test of each type of selector to check parsing and functionality. Also includes some fixes
discovered while creating unit tests

- Update existing selectors so they are current with the specification.
- Also add early implementations of ExploreIndex and ExploreRange.
- Add an early implementation of Explore for ExploreUnion.
- Add constants for field keys.

The Go Playground is also often a perfectly good home for these things;
I've also used "_rsrch" directories in other projects, especially if
benchmarks of nontrivial runtime and setup get involved; but this
will suffice for this quick factcheck here and now.

I'm solving the decorum problem by making "temporary" builder methods for the
schema.Type values as necessary.  This will allow the gengo package to use
them freely for now.  When we're done with all this, I want to restrict building
of schema.Type values to a single method in the schema package which takes the
"ast" types -- the ones we're hoping to codegen (!) -- and do lots of checks
and cross-linking; the temporary builder methods will do *none*, and *of course*
they can't take the "ast" types yet since they don't exist yet.

Boom, cycle broken.

Also: fix generateStringKind from taking a TypeName redundantly.

The schema.Type objects don't have a `Kind() ReprKind` method.  That
was just a bad idea.  I didn't try to fix it; I just nuked it.

The new schema.Type.Kind() method docs include a description of why
it's incorrect to wish for a either a single or a statically known
ReprKind for a type.

(Context mention: the previous commit was also a wind-up for this.
And technically, I'm still winding back up the mental stack here:
this had to get straightened out because I wanted to start poking
codegen for structs; and then I had beef with the TypeStruct.tupleStyle
field being a bool; and then fixing that pointed out that the current
Type.ReprKind methods were kinda insane... yaks!)

Comments added addressing some of the root issues of why, and how to
try again more correctly next time.  (At least one open question
has been found that might take a few days to resolve.)

This commit happens now because they code being removed refered to
`schema.Type.ReprKind()` a lot, and that's the real issue I'm focused
on at the moment -- that's also going away in the next commit or so.
There's significantly more subtlety to the type-vs-representation
duality than the existence of such a method could accurately describe.