GitLab

Signed-off-by: Eric Myhre <hash@exultant.us>

We now have both MapIterator and ListIterator interfaces.
Both return key-value (or index-value) pairs, rather than just keys.

List iterators may seem a tad redundant: you just loop over the length,
right?  Well, sure.  But there's one place a list iterator shines:
selecting only a subset of elements.  And indeed, we'll be doing
exactly that in the traversal/selector package; therefore, we
definitely need list iterators.

We might want keys-only iterators again in the future, but at present,
I'm deferring that.  It's definitely true that we should have iterators
returning values as a core feature, since they're likely to be more
efficiently supportable than "random" access (especially when we get to
some Advanced Layout data systems), so we'll implement those first.

Additionally, note that MapIterator now returns a Node for the key.
This is to account for that fact that when using the schema system and
typed nodes, map keys can be more *specific* types.  Such nodes are
still required to be kind==ReprKind_String, but string might not be
their *preferred* native format (think: tuples with serialized to be
delimiter-separated strings); we need to account for that.
(MapBuilder.Insert method already takes a Node parameter for similar
reasons: so it can take *typed* nodes.  Node.TraverseField accepting
a plain string is the oddball out here, and should be rectified.)
Signed-off-by: Eric Myhre <hash@exultant.us>

Having a function called "Kind" return a "ReprKind" was inconsistent.

Also, we want to introduce a "Kind" method on `typed.Node` in the future.

No logical content to this change: you can safely refactor with sed.
Signed-off-by: Eric Myhre <hash@exultant.us>

(... offically.  Lots of docs have probably already been stating that
this is there.  Now it actually... is.)
Signed-off-by: Eric Myhre <hash@exultant.us>

This has been deprecated and replaced by the NodeBuilder system
for a good while now; time to scrape it into the dustbin completely.

Tests that were primarily on the mutable node system itself also
drop, so, this is a *very* large delete diff.

A few other tests used MutableNode just incidentally, and those are
quick fixed to use NodeBuilder.
Signed-off-by: Eric Myhre <hash@exultant.us>

As detailed in comments a few commits ago, this is part of a big, big
roll towards keeping linking details far enough off to one side that
one can actually use most of the IPLD system without forming an
explicit compile-time dependency on any linking features (until, of
course, one uses the linking features).

This is a surprisingly small diff, because... well, because most of
the *interesting* features around linking simply weren't implemented
yet, and at this point everything that is has already been isolated
in the new cidlink and related encoding packages.
"CID" was *already* just a semantic placeholder that meant "eh, link".
Signed-off-by: Eric Myhre <hash@exultant.us>

We have both generic marshal and unmarshal -- they should work for any
current or future ipld.Node implementation, and for any encoding
mechanism that can be bridged to via refmt tokens.

Tests are also updated to use builders rather than the ancient
"mutable node" nonsense, which removes... I think nearly the last
incident of that stuff; maybe we can remove it entirely soon.

As when we moved the unmarshal code into its generic form, most of
this code already existed and needed minor modification.  Git even
correctly detects it as a rename this time since the diff is so small.
And as when we moved the unmarshal code, now we also remove the
whole PushTokens interface; we've gotten to something better now.

Finally we're getting to the point we can look at wiring these up
together with all the multicodec glue and get link loading wizardry
at full voltage.  Yesss.  Sooon.
Signed-off-by: Eric Myhre <hash@exultant.us>

This unmarshal works for any NodeBuilder implementation, tada!

Old ipldfree.Node-specific unmarshal dropped... as well as that entire
system of interfaces.  They were first-pass stuff, and I think now it's
pretty clear that it was barking up the wrong tree, and we've got better
ideas to go with now instead.  (Also, as is probably obvious from a skim,
the old code flipped pretty clearly into the new code.)

Turns out refmt tokens aren't a very relevant interface in IPLD.
I'm still using them... internally, to wire up the CBOR and JSON
parsers without writing those again.  But the rest of IPLD is more
like a full-on and principled alternative to refmt/obj and all its
reflection code, and that's... pretty great.

Earlier, I had a suspicion that we would want more interfaces for token
handling on each Node implementation directly, and in particular I
suspected we might use those token-based interfaces for doing transcription
features that flip data from one concrete Node implementation into another.
(That's why we had this ipldfree.Node-specialized impl in the first place.)
**This turns out to have been wrong!**  Instead, now that we have the
ipld.NodeBuilder interface standard, that turns out to be much better suited
to solving the same needs, and it can do so:

- without adding tokens to the picture (simpler),

- without requiring tokenization-based interfaces be implemented per
concrete ipld.Node implementation (OH so much simpler),

- and arguably NodeBuilder is even doing it *better* because it doesn't
need to force linearization (and while usually that doesn't matter... one
can perhaps imagine it coming up if we wanted to do a data transcription
in memory into a Node implementation which has an orderings requirement).

So yeah, this is a nice thing to have been wrong about.  Much simpler now.

Old ipldfree.Node-specialized 'PushTokens' is still around.  Not for long,
though; it just hasn't finished being ported to the new properly generalized
style quite yet.

Note, this is not the *whole* story, still, either.  For example, still
expect to have an ipldcbor.Node which someday has a *significantly* different
set of marshal and unmarshal methods -- it may eschew refmt entirely,
and take the (very) different strategy of building a skiplist over raw
byte slices! -- that will exist *in addition* to the generic implementations
we're doing here and now.  More on that soon.

Yeah.  A lot of interfaces to get lined up, here.  Some of them tug in such
different directions that picking the right ones to make it all possible
seems roughly like solving one of the NP-hard satisfiability problems.
(Good thing it's actually with a small enough number of choices that it's
tractable; on the other hand, enumerating those choices isn't fast, and
the 'verifier' function here ain't fast either, and being a "design" thing,
it can only be evaluated on human wetware.  So yeah, an NP problem on a
tractable domain but slow setup and slow verifier.  Sounds about right.)

(uh, I'm going to write a book "Design: It's Hard: The Novel" after this.)

Tests are patched enough to keep working where they are; I think it's
possible that a reshuffle of some of them to be more closely focused on
the marshal code rather than the node implementation packages might be
in order, but I'm going to let that be a future issue.  (Oh, and they
did shine a light on one quick issue about MapBuilder initialization,
which is also now fixed.)
Signed-off-by: Eric Myhre <hash@exultant.us>

This is the first commit going down a long and somewhat dizzying prerequisite tree:

- For graphsync (an out-of-repo consuming project) we need selectors
- For Selectors we need traversal implemented
- For Traversal implementations we need link loaders [‡]
- For link loading we need all deserialization implemented
- (and ideally, link creation is done at the same time, so we don't get surprised by any issues with the duals later)
- and it turns out for deserialization, we now have some incongruities with the earlier draft at MapBuilder...

So we're all the way at bugfixes in the core ipld.MapBuilder API.  Nice.

([‡] Some of those jumps are a little strained.  In particular, traversal doesn't
*in general* need link loaders, so we might choose a very different implementation
order (probably one that involves me having a lot less headaches)... *except*,
since our overall driver for implementation order choices right now is graphsync,
we particularly need traversals crossing block boundaries since we're
interested in making sure selectors do what graphsync needs.  Uuf.)

What's the MapBuilder design issue?  Well, not enough error returns, mostly.
We tried to put the fluent call-chaining API in the wrong place.

Why is this suddenly an issue now?  Well, it turns out that properly genericising
the deserialization needs to be able to report error states like invalid
repeated map keys promptly.

Wait, didn't we even *have* deserialization code before?  Yes, yes we did.
It's just that that draft was specialized to the ipldfree.Node implementation...
and so it doesn't hold up anymore when we need it to work in general traversal.

Okay, so.  That's the stack depth.

With all that in mind...

This diff adds more error return points to ipld.MapBuilder, and maintains the
fluent variant more closely matching the old behavior (including the
call-chaining style), and fixes tests that relied on this syntax.

Duplicate keys rejection is also in this commit.  I thought about splitting it
into further commits, but it's so small.  (We may actually need more work in
the future to enable Amend+(updating)Insert, but that's for later; perhaps
an Upsert method; whatever, I dunno, out of scope for thought at the moment.)

And then we'll carry on, one step at a time, from there.  Whew.

---

Sidebar: also dropping MapBuilder.InsertAll(map[Node]Node) from the interface.
I think this could be better implemented as a functional feature that works
over a MapBuilder than being a builtin, and we should prefer a trim MapBuilder.
And might as well drop it now rather than bother fixing it up just to remove later.

---

ipld.ListBuilder also updated to no longer do a call-chaining style API, while
fluent.ListBuilder continues to do so.  This is mainly for consistency;
we don't have the same potential for mid-build error conditions for lists
as we do with maps, but ipld.ListBuilder and ipld.MapBuilder should be similar.

---

Aaaaand one more!  NodeBuilder.{Create,Append}{Map,List}() have ALL been
updated to also return errors.  Previously, the append methods had an error
state if you used them when the NodeBuilder was bound to a predecessor node
of an unmatching type, but they just swallowed them into the builder and
regurgitated them (much) later; we're no longer doing this.  Additionally,
it's occurred to me that *typed* builders -- while not so much a thing, yet,
certainly a thing that's coming -- will even potentially error on CreateMap
and CreateList methods, according to their type constraints.  So, jump that now.

...

Yeah, basically a whole tangle of misplaced optimism about error paths in
NodeBuilder and its whole set of siblings has been torn through at once here.
Bandaid ripping sound.
Signed-off-by: Eric Myhre <hash@exultant.us>

We now have nice, immutable nodes... as soon as we deprecate and remove
the MutableNode stuff.

All the tests pass.

A few methods for bulk update are still stubbed... as is delete;
seealso the comment about that in the API file.

NodeBuilders needed replication in the fluent package as well.
This is perhaps a tad frustrating, as well as borderline surprising
(because "how can sheer memset have errors anyway?"), but the ability
for node building to reject invalid calls will become relevant when
typed.Node is in play and has to enforce type rules while still
supporting the generic methods.

The builder syntax for maps and lists is based on chaining.
This works... okay.  I'm not as satisfied with the visual effect of it
as I'd hoped.  There'll probably be more experiments in this area.
Things nest -- critical success; but the indentation level doesn't
quite match what we'd contemporarily consider natural flow.

But look at the diff between the mutableNode test specs and the new
immutable&fluent nodeBuilder test specs!  The old ones were nuts:
temp variables everywhere for construction, total visual disorder, etc.
The new NodeBuilder is much much better for composition, and doesn't
need named variables to save fragments during construction.
More refinement on map/list builders still seems possible, but it's
moving in the right direction.
Signed-off-by: Eric Myhre <hash@exultant.us>

Allows uncommenting some fixme's in some tests.

Also, changed free.keyIterator to be unexported.  Exporting that was
a typo; there's simply no reason for that to be exported.
Signed-off-by: Eric Myhre <hash@exultant.us>

Hej, we can finally unmarshal things again.  Plug a cbor parser
in from the refmt library and go go go!

This doesn't use any of the mutablenode stuff because, as remarked
upon a (quite a) few commits back, we want to replace that with
NodeBuilder interfaces; but, furthermore, we actually don't *need*
those interfaces for anything in an unmarshalling path (because
I don't expect we'll seriously need to switch result Node impl
in mid-unmarshalling stream); so.

You can imagine that the cbor.Node system will have a *very* different
implementation of this interface (and probably another method that
doesn't match this interface at all, and is more directly byte stream
based); but that's for later work.

And tests!
Signed-off-by: Eric Myhre <hash@exultant.us>

An "immediate" rather than generative function is also available;
the docs contain caveats.

At the moment, these distinctions are a bit forced, but we want to be
ready for when we start getting truly huge maps where the generative
usage actually *matters* for either memory reasons, latency reasons,
or both.

Separated Length rather than trying to pull double-duty the Keys
method; the previous combination was just utterly silly.

The implementations in the bind package are stubs; that package is
going to take a lot of work, and so the focus is going to be entirely
on keeping the 'free' package viable; and we'll revisit bind and such
when there's more dev time budget.
Signed-off-by: Eric Myhre <hash@exultant.us>

Signed-off-by: Eric Myhre <hash@exultant.us>

It should return an error explicitly.

A pair of "(nil, nil)" reponses would also be an unambiguous option for
representing an absent member, but seems more likely to create pitfalls
than save a significant amount of code for callers.
Signed-off-by: Eric Myhre <hash@exultant.us>

You could now readily rig up the ipldfree.Node implementation to a
refmt cbor.TokenSink, for example, and go to town.  (At the moment,
doing so is left as an exercise to the reader.  We'll make that a
smooth built-in/one-word function at some point, but I'm not yet sure
exactly how that should look, so it's deferred for now.)

While working on this, a lot of other things are cooking on simmer:
I'm churning repeatedly over a lot of thoughts about A) API semantics
in general, B) how to cache CIDs of nodes, and C) how to memoize
serializations / reduce memcopies during partial tree updates...
And (unsurprisingly) I keep coming back to the conclusion that the API
for dang near everything should be immutable at heart in order to
keep things sane.  The problem is figuring out how to pursue this
A) efficiently, B) in tandem with reasonably low-friction nativeness
(i.e. I want autocompletion in a standard golang editor to be as useful
as possible!), and C) given an (as yet) lack of good builder or
mutation-applier patterns.  ipldbind was meant to be a solution to the
majority of the B and C issues there, but that rubs smack against the
grain of "let's be immutable" in golang >:/  So... a rock and a hard
place, in short.
Signed-off-by: Eric Myhre <hash@exultant.us>

These are all skullduggerish, but basic sanity checks are nice when
you've got >3 impls of a thing to keep an eye on.
Signed-off-by: Eric Myhre <hash@exultant.us>

ipldfree and ipldbound and fluent and all the interfaces now have all
the isNull methods, asBytes, setBytes, etc.  (Previously this was a
little swiss-cheesy.)

SMOP.
Signed-off-by: Eric Myhre <hash@exultant.us>

It's now "invalid" rather than A) documented as map, and B) actually
panicky.
Signed-off-by: Eric Myhre <hash@exultant.us>

This will be for the active path -- if we also follow through on the
idea of having a just-in-time checked Node wrapper, a lot of these
checks might end up duplicated.  We'll DRY that up when we get there.

Doing accumulation of errors.  Might get loud.  We can come back and
and early-halt parameters or other accumulation strategies later.

Added IsNull predicate to core Node interface.

Going to want enumerated error categories here for sure, but punting
on that until we get more examples of what all the errors *are*; then
I'll come back and tear all this 'fmt.Errorf' nonsense back out.
Signed-off-by: Eric Myhre <hash@exultant.us>

And ReprKind moves from the typed package to the ipld main package.
It's hard to get too much done without the standardization of ReprKind.

Between the Kind() and Keys() methods, it should now be possible to
perform traversals of unknown nodes.

This diff just worries about implementing all the Kind() methods.
Keys() has some additional questions to handle (namely, map ordering?).
Signed-off-by: Eric Myhre <hash@exultant.us>

And a test of building an array and pulling a value back out.
Signed-off-by: Eric Myhre <hash@exultant.us>

Since this uses a MutableNode factory method as a parameter, this test
is *generic* over our implementations.

Woot!

(Also, used the fluent wrapper interface to make things less
syntactically irritating on the reading side.)
Signed-off-by: Eric Myhre <hash@exultant.us>

(Mostly; all the scalars work; the composites are todo's.)

Heading in the direction of being able to construct stuff for testing.
Almost there.
Signed-off-by: Eric Myhre <hash@exultant.us>

Not sure why these weren't there already.

(Probably because of the ongoing discussion about "is link a kind at
the data model level?" -- to which I still opine "ideally, no".  But
that's something we're probably going to concede "yes" on *for now*,
and try to straighten out later once we get something ready for the
higher level type systems.)
Signed-off-by: Eric Myhre <hash@exultant.us>

Sort float and bytes (less ubiquitous things) to the bottom.

Drop mention of uint from the ipldfree implementation.  So far all spec
discussions have tended towards mentioning "integers" as a type, and
definitely not "unsigned integers" or "positive-only integers" as a
distinct kind in the core Data Model.
Signed-off-by: Eric Myhre <hash@exultant.us>

Finally got the bind and free impls on the same page.

Surprisingly, the bind node can work without being a ptr itself.
I'm not sure if that'll last, but let's try to roll with it.
If we *can* keep that property, it might reduce GC impact in a pretty
significant way.

Added a 'fluent' package.  It's just a twinkle in my eye so far, but it
might represent the nicest way through all the discussed issues.
Nodes shouldn't have to be panicful; and users shouldn't have to do all
error handling manually either.  A package full of fluent interfaces
that know what's going on seems to be the only way to get both.
But we'll see how this shakes out.  Maybe typeful traversers will
make the whole thing end up more coreward than being relegated to a
helper package.  I have no idea.
Signed-off-by: Eric Myhre <hash@exultant.us>

Some comments on the interface as well.  Almost certain that traversal
should be pulled off the node interface itself.
Signed-off-by: Eric Myhre <hash@exultant.us>