GitLab

- `typed.Node` -> now lives in the `impl/typed` package, more like
other nodes.

- Most of the other essential parts of reasoning about types moved
to `schema` package.  ("typed.Type" seemed like a nasty stutter.)

- `typed.Universe` renamed to `schema.TypeSystem`.

- Current `Validate` method moved to `schema` package, but see new
comments about potential future homes of that code, and its
aspirational relationship to `typed.Node`.

Conspicuously *not* yet refactored or moved in this comment:

- The `typed/declaration` package -- though it will shortly be scrapped
and later reappear as `schema/ast`.  The dream is that it would be
neatest of all if we could generate it by codegen; but we'll see.
(This would seem to imply we'll have to make sufficient exported
methods for creating the `schema.Type` values... while we also want
to make those immutable.  We'll... see.)

- The `typed/gen` package is also untouched in this commit, but
should similarly move shortly.  The codegen really ought to be built
against the `schema.Type` reified interfaces.

Overall, this drops the sheer nesting depths of packages a fair bit,
which seems likely to be a good smell.

These are now going to go live in the ipld/specs repo.

(And @vmx has done the git-fu to even keep their history intact over
in their new home, which is awesome.)

Onward!

Cherry-picked by eric with several changes.

- The fixups of missing bits of schema-schema are a fair cop.

- Several critical typos in the example.ipldsch.json file are fixed.

- I tweaked the syntax for aliases in the example file; we've had the
  idea since this was originally posted of putting representational
  concerns that are clearly associated with a field inline with the
  fields for convenience, but in parens to distinguish that they're
  indeed representation concerns rather than type/cardinality concerns,
  so this is the expression of that idea.

- I actually walked back the removal of representation blocks in the
  json.  There's probably valid considerations to be made on that, but
  it deserves separate handling.  (Disconcertingly, I actually added
  two *new* representation blocks that seem to previously have been
  missing due to typos.)

Any new errors are presumably mine.

Original: https://github.com/ipld/go-ipld-prime/pull/10

The underscore makes it ignored by the go tools;
the gitignore does, well, git.

At some point this should probably grow up to have more complete
tooling and test suites around it, but for early iteration,
I pretty much have one window open to the generated output file,
and I'm just running it to see if it compiles, and while this probably
won't last for long, it's enough to get exploratory work done.

In thinking about how to make a 'bind' (aka, reflect and atlases) Node
implementation, some interesting stuff comes up: despite being all
one concrete Node implementation, it needs specialization (for the
reflect.Value bound inside, specifically); and 'bind' nodes don't
*necessarily* have a schema and a typed.Node associated with them,
so the existing comments about specializing using Type info don't
actually apply.  So!  What do?

These comments are a tad hypothetical (and have been on my uncommitted
working tree for a while, so hopefully they're not *too* stale; I just
want to get them in history somewhere rather than keep dancing my
patches around them)... but there's almost certainly something to
address somewhere in this area.

Begin Selector implementation!

Noticed while reviewing the parent commit, so double thanks for that one...!

Fix doc

The iteration can stop if the `Done()` method returns true, not false.

This is a draft.  There's a LOT of hard design choices which are not
yet completely settled here.

Corresponding updates to the ipld/specs repo will be coming soon:
these implementations match a very recursive concept of selectors.
(https://github.com/ipld/specs/pull/116 is roughly this.)

There's so many things to go over, here.

- selectors at all
- selectFields is kind of a special case of unions
- selectPath as a concept is gone; is a degenerate case of selectFields
- we've added some new methods to help narrow down search spaces
- we have serial selector spec parsers...!  (very hand-rolled.)
- the idea of always reporting candidates vs definite match parents
  is dropped for now; it requires a different algo with different
  performance characteristics, so, we'll do it as such -- later.
- we ended up with unions being *in*!  They're going to be used
  internally by a bunch of other things -- see comment blocks in
  the code about that (tl;dr recursives imply a need of unions).

More docs will be coming in future commits; this is just a checkpoint
of the progress.
Signed-off-by: Eric Myhre <hash@exultant.us>

The previous code was lots of switches; I think it's safe enough to
say that wasn't going to scale or compose very easily.
This new take is based on some interfaces and a wee dusting of
polymorphism.

The abstraction *won't* hold: we'll drill through it in many places,
and there will still be type switches up the wazoo by the end -- see
the comments already scattered about regarding maps and enums and such.
But that's okay; a few interfaces will still help; at the very least
it makes things a bit more structurally self-documenting.

Strings (and one particular representation and node implementation of
them) are all that's included here, but so far so good.

The comment above, and the one in the code about "triple cross
product", probably waggle towards what's going to be the trickiest
part of this: codegen needs to take into account a *lot* of choices.
Making the code for this maintainable is going to be nontrivial!
Signed-off-by: Eric Myhre <hash@exultant.us>

(n.b. like a couple other things I'm about to finally commit, this has
been a floating diff in my working copy for a while already now;
it might be a slightly stale understanding of something; but it's
a part of history now, anyway, whether or not we move on from it.)

(These have been on my working copy for *months*, waiting to get some
sliver of time to get back to working on this half of the project...)

So, back in the last time period I was working on schemas, it seemed
plausible to try to get some self-bootstrapping amount of codegen to
help me write the whole declaration package itself.  This has not
held up, lol.

The long end of line comment is my updated feelings about this:
the main reason self-hosting even during bootstrap would've been nice
is for map order consistency.  But... we don't... need that.  Per se.

This commit will be merge-ignored momentarily; gonna pick up on
schema work again starting from a somewhat different angle of attack.
Signed-off-by: Eric Myhre <hash@exultant.us>

And test.
Signed-off-by: Eric Myhre <hash@exultant.us>

Autocomplete on TraversalProgress was suggesting *way* too many things.
It's now reigned in.

Being able to say `tp.Ctx` is gone, but that's a small price to pay.
(And perhaps we should actually move the Context up to TP?  Wouldn't
seem unreasonable.  I think the main reason it's in TC right now is
to reduce the size of memory that's shuffled when TP is passed by value
down the stack, and it's unclear how consequential that's really
going to be in the big picture.  We can review this later.)
Signed-off-by: Eric Myhre <hash@exultant.us>

NodeBuilderChooser returned -- sorry.  There are some cases where it's
unavoidable.  (If we have schemas / typed nodes, we can do feature
detection on the node to see if it can recommend an appropriate
NodeBuilder for the far side of its link.  If we're fine using Node
implementations that have 'any' storage support, it's not an issue.
If we're using *bind* implementation Nodes -- which are constrained
by the Golang native type system, and yet not perfectly mapped onto
our own typed.Node constraint declarations -- then we need some way
to choose the NodeBuilder.  So.  NodeBuilderChooser.  Again: sorry.)

We can probably still expect the NodeBuilderChooser to be fixed
(probably returning one of the 'any' storage supporting Node impls)
in practice, since these traversal systems are generally for either
"low context" usage (e.g., dunno what the data is) or will be used
on typed/schema nodes, in which case we're again free of these issues.
And the default values will indeed do this, using our new helpful
default initialization for TraversalConfig.

TraversalConfig will now always be initialized to sensible default
values when using any of the traversal methods.  That means you can
always assume `tp.Ctx`, etc, are going to be set -- no nil checks
necssary -- even though they're optional to the caller.

Error messages from traversal.Focus are touched up a bit for both
consistency and clearer information.  There's more work to do here:
we want these to be full-on typed errors before we call it "done".
This work will probably begin soon, but be spread over quite a few
commits (we also need to go back up to the *main* ipld package and
make types for Node-level errors, and make sure tests exist to
standardize those errors across Node implementations as well).
Signed-off-by: Eric Myhre <hash@exultant.us>

Emitting just the multihash part, though in some ways technically
correct, is really weird looking.

Let's use the CID consistently in messaging.
Signed-off-by: Eric Myhre <hash@exultant.us>

Signed-off-by: Eric Myhre <hash@exultant.us>

Including one interesting fix for dagjson.

Since json can include whitespace -- and especially since our
implementation currently uses prettyprinted json with quite a bit of
said whitespace -- it's important to handle it consistently.

We had a fun issue here: the json would be emitted with a trailing
linebreak (as is generally what you want for printing to a terminal,
etc!)... and thus hashed with it.  Then, when loading the object,
our parser will load exactly every byte needed to parse the object,
then stop.  Which... will cause it to return right before consuming
that trailing linebreak.

Which would cause that trailing linebreak to not be fed into the
hasher, since we've carefully used a system which tees exactly the
bytes consumed by the parser into the the hasher.

So of course the link hash validation would fail.  Woowee.

I documented some of these details in an issue on the specs repo:
https://github.com/ipld/specs/issues/108
There's not a super clear resolution over there as yet, but there seems
to be a general agreement that whitespace should be tolerated, so...
let's do so.

As of this patch, the dagjson unmarshaller will consume all additional
whitespace after finishing consumption of the json object iself.

Dagcbor doesn't need a similar fix: there's no such thing as any
possibility of other nonsemantic bytes, so there's nothing to absorb;
and if we don't reach the end of the reader... we technically don't
*care*: given the same reader over the same data being used to load
the same link, we'll behave consistently; and therefore it follows that
any additional bytes in the reader are unobservable to our universe.

An earlier (and badly broken) draft of this attempted to put the
read-to-end behavior in the cidlink package, but in addition to being
unnecessary for dagcbor as described above, it also would've been
simply *wrong*: the whitespace slurp is specific to dagjson.
Signed-off-by: Eric Myhre <hash@exultant.us>

The refmt step functions expose "done" a little more eagerly than this
code figured on, so that's been corrected.
Signed-off-by: Eric Myhre <hash@exultant.us>

Has some readability improvements, and a substantial jump forward in
the vendored go-cmp library.

No major functional changes; just pretties.
Signed-off-by: Eric Myhre <hash@exultant.us>

Node contained in MapBuilder and LinkBuilder don't need another separate pointer and heap allocation.
Signed-off-by: Eric Myhre <hash@exultant.us>

Signed-off-by: Eric Myhre <hash@exultant.us>

We now have both MapIterator and ListIterator interfaces.
Both return key-value (or index-value) pairs, rather than just keys.

List iterators may seem a tad redundant: you just loop over the length,
right?  Well, sure.  But there's one place a list iterator shines:
selecting only a subset of elements.  And indeed, we'll be doing
exactly that in the traversal/selector package; therefore, we
definitely need list iterators.

We might want keys-only iterators again in the future, but at present,
I'm deferring that.  It's definitely true that we should have iterators
returning values as a core feature, since they're likely to be more
efficiently supportable than "random" access (especially when we get to
some Advanced Layout data systems), so we'll implement those first.

Additionally, note that MapIterator now returns a Node for the key.
This is to account for that fact that when using the schema system and
typed nodes, map keys can be more *specific* types.  Such nodes are
still required to be kind==ReprKind_String, but string might not be
their *preferred* native format (think: tuples with serialized to be
delimiter-separated strings); we need to account for that.
(MapBuilder.Insert method already takes a Node parameter for similar
reasons: so it can take *typed* nodes.  Node.TraverseField accepting
a plain string is the oddball out here, and should be rectified.)
Signed-off-by: Eric Myhre <hash@exultant.us>

Naming: ReprKind.

fix(cidlink): init multicodec registry

allocate multicode tables so register functions don't panic

Having a function called "Kind" return a "ReprKind" was inconsistent.

Also, we want to introduce a "Kind" method on `typed.Node` in the future.

No logical content to this change: you can safely refactor with sed.
Signed-off-by: Eric Myhre <hash@exultant.us>

Thanks Hannah!
Signed-off-by: Eric Myhre <hash@exultant.us>

(... offically.  Lots of docs have probably already been stating that
this is there.  Now it actually... is.)
Signed-off-by: Eric Myhre <hash@exultant.us>

This has been deprecated and replaced by the NodeBuilder system
for a good while now; time to scrape it into the dustbin completely.

Tests that were primarily on the mutable node system itself also
drop, so, this is a *very* large delete diff.

A few other tests used MutableNode just incidentally, and those are
quick fixed to use NodeBuilder.
Signed-off-by: Eric Myhre <hash@exultant.us>