GitLab

Signed-off-by: Eric Myhre <hash@exultant.us>

...and all fields are correspondingly unexported.

We're going full immutability on almost all this stuff.  I simply can't
imagine handling and maintaining this any other way.

The typesystem.Type interface also got a few more standard methods.
I'm not sure if having a Universe backpointer is going to be useful
very often or not, but it seems reasonable enough to do; we'll
definitely *have* that information during consistency checking and
reification time, so it should turn out easy to keep it.

The validate_test.go code is now looking a bit janky.  We need
initializers for the typesystem.Type* values.  Those will be the
typedeclaration.* package... which we'll write next.

Note that I skipped any accessor methods on `Type*` objects which are
for representation details.  These might get filled in later, but I'm
gonna see how far I can get without them.  (So far, there's some
uses of the representation details in the validate method, but we've
located that in the typesystem package anyway, so that's... fine.)
Signed-off-by: Eric Myhre <hash@exultant.us>

See previous commit for more discussion of the child packages to come,
or the schema.md file in the docs dir, which describes similar.

Several large todos in comments.

Quite importantly: typed.Node is now an **interface**, not a concrete
struct itself.  The godoc comment should explain this nicely; long
story short, this is going to be relevant when we get to codegen and
other advanced form of native integration.
Signed-off-by: Eric Myhre <hash@exultant.us>

I need there to be separate packages for:

- the typed.Node interface, and any top level helpers;
- the typesystem implementation
- the typesystem *declarations* (distinct because it's what you
 construct when parsing the ast -- doens't contain pointers yet)
- and (eventually) a parser and a fmt package for the ts declarations.

And right now all those are heaped together.

Not sure where the validate method -- the thing that's actually touched
in this diff -- will end up.  Either the top level, or the typesystem
implementation package, probably.
Signed-off-by: Eric Myhre <hash@exultant.us>

You could now readily rig up the ipldfree.Node implementation to a
refmt cbor.TokenSink, for example, and go to town.  (At the moment,
doing so is left as an exercise to the reader.  We'll make that a
smooth built-in/one-word function at some point, but I'm not yet sure
exactly how that should look, so it's deferred for now.)

While working on this, a lot of other things are cooking on simmer:
I'm churning repeatedly over a lot of thoughts about A) API semantics
in general, B) how to cache CIDs of nodes, and C) how to memoize
serializations / reduce memcopies during partial tree updates...
And (unsurprisingly) I keep coming back to the conclusion that the API
for dang near everything should be immutable at heart in order to
keep things sane.  The problem is figuring out how to pursue this
A) efficiently, B) in tandem with reasonably low-friction nativeness
(i.e. I want autocompletion in a standard golang editor to be as useful
as possible!), and C) given an (as yet) lack of good builder or
mutation-applier patterns.  ipldbind was meant to be a solution to the
majority of the B and C issues there, but that rubs smack against the
grain of "let's be immutable" in golang >:/  So... a rock and a hard
place, in short.
Signed-off-by: Eric Myhre <hash@exultant.us>

These are all skullduggerish, but basic sanity checks are nice when
you've got >3 impls of a thing to keep an eye on.
Signed-off-by: Eric Myhre <hash@exultant.us>

Signed-off-by: Eric Myhre <hash@exultant.us>

Signed-off-by: Eric Myhre <hash@exultant.us>

Signed-off-by: Eric Myhre <hash@exultant.us>

ipldfree and ipldbound and fluent and all the interfaces now have all
the isNull methods, asBytes, setBytes, etc.  (Previously this was a
little swiss-cheesy.)

SMOP.
Signed-off-by: Eric Myhre <hash@exultant.us>

Used the ipldfree.Node impl concretely to make the tests work.
In theory any impl should of course work, but it's not useful or
interesting to do a compatibility matrix here -- particularly given
that we use the fluent system in the compataibility tests for all
the other Node impl packages.
Signed-off-by: Eric Myhre <hash@exultant.us>

Fairly sure this is going to come up in our handling of typed nodes
(and may also be relevant to some of the REVIEW-todo comments about
sparseness of arrays; haven't yet fully considered).
Signed-off-by: Eric Myhre <hash@exultant.us>

It's now "invalid" rather than A) documented as map, and B) actually
panicky.
Signed-off-by: Eric Myhre <hash@exultant.us>

Signed-off-by: Eric Myhre <hash@exultant.us>

Contents:

- A few high level things, mostly just to give enough minimal context and
terminology to continue, and shelling out to the ipld/specs repo for
details.

- "Everything is a Node"

- Code layout with brief words about the intent behind each tree of
packages.

One of the main things I wanted to get on paper this morning is the
distinction between the reified/active/usable type-checker types,
versus the tree that represents them serially.  The former may contain
cyclic references; the latter obviously cannot.  We don't yet have
separate packages and types for these, but should asap.  This
description all snuck in under "code layout".
Signed-off-by: Eric Myhre <hash@exultant.us>

This will be for the active path -- if we also follow through on the
idea of having a just-in-time checked Node wrapper, a lot of these
checks might end up duplicated.  We'll DRY that up when we get there.

Doing accumulation of errors.  Might get loud.  We can come back and
and early-halt parameters or other accumulation strategies later.

Added IsNull predicate to core Node interface.

Going to want enumerated error categories here for sure, but punting
on that until we get more examples of what all the errors *are*; then
I'll come back and tear all this 'fmt.Errorf' nonsense back out.
Signed-off-by: Eric Myhre <hash@exultant.us>

And ReprKind moves from the typed package to the ipld main package.
It's hard to get too much done without the standardization of ReprKind.

Between the Kind() and Keys() methods, it should now be possible to
perform traversals of unknown nodes.

This diff just worries about implementing all the Kind() methods.
Keys() has some additional questions to handle (namely, map ordering?).
Signed-off-by: Eric Myhre <hash@exultant.us>

Having a Name() interface method might also have been useful, but at
the moment, it's too annoying.  Maybe I'll come back to this and add a
ton of constructors for each kind of type and make all their fields
private scope, which would solve the name collision... maybe.  Later.
(This is all expected to be implementation-internal stuff in the long
run rather than anything user-facing API, so it's up for debate how
much polishing it's actually worth.  Unless that changes!)

The ReprKind method is *usually* pretty predestined based on the kind
of type in the first place, but a few cases are interesting.

... *Especially* kinded unions.  I'm a little alarmed at the break of
pattern, there.  Hopefully that doesn't manifest too much complexity
down the road.  But if it does... eh, well... nature of the beast.
Kinded unions are definitely a useful feature.
Signed-off-by: Eric Myhre <hash@exultant.us>

There's... not really much to it.
Signed-off-by: Eric Myhre <hash@exultant.us>

Signed-off-by: Eric Myhre <hash@exultant.us>

Giving a type to each, um, kind of type.

Syntax draft changed.  Bigger examples elsewhere.  May need to start
writing up more, but might not belong in this repo, either (it's not
a detail of the golang implementation alone).  Anyway, it's only for
eyecandy at present: I'm not going to start work on a parser for the
DSL any time soon; it's just something to keep in mind so it can be
evaluated against simplicity heuristics.

Struct fields are in a slice now.

Waffling on whether to call that kind "struct" vs "object".  Neither is
very satisfying.  "Struct" is suggestive of something with concrete
memory layout considerings; this is not that.  "Object" is suggestive
of something with methods and behaviors attached; this is not that.

Introduce an enum for different kinds of union.  Some of the terms here
are developed from https://github.com/ipfs/unixfs-v2/issues/20 ; the
term "keyed" is from https://github.com/polydawn/refmt/pull/30 ; and
the style "kinded" is a concept I haven't written up much about before,
but the TypeUnion.ValuesKinded field type should self-explain it.

Flip terminology from "required" to "optional".  Struct fields that are
mentioned in the schema default to being required; why else would you
have mentioned them?  Previous draft of DSL which used question marks
and exclamations points to denote these things dropped; the new hope is
that *neither* nullable nor optional will be used with enough frequency
that the sigil-heavy shorthand will turn out to be desirable.
Signed-off-by: Eric Myhre <hash@exultant.us>

And a test of building an array and pulling a value back out.
Signed-off-by: Eric Myhre <hash@exultant.us>

Since this uses a MutableNode factory method as a parameter, this test
is *generic* over our implementations.

Woot!

(Also, used the fluent wrapper interface to make things less
syntactically irritating on the reading side.)
Signed-off-by: Eric Myhre <hash@exultant.us>

Signed-off-by: Eric Myhre <hash@exultant.us>

(Mostly; all the scalars work; the composites are todo's.)

Heading in the direction of being able to construct stuff for testing.
Almost there.
Signed-off-by: Eric Myhre <hash@exultant.us>

Not sure why these weren't there already.

(Probably because of the ongoing discussion about "is link a kind at
the data model level?" -- to which I still opine "ideally, no".  But
that's something we're probably going to concede "yes" on *for now*,
and try to straighten out later once we get something ready for the
higher level type systems.)
Signed-off-by: Eric Myhre <hash@exultant.us>

Signed-off-by: Eric Myhre <hash@exultant.us>

Sort float and bytes (less ubiquitous things) to the bottom.

Drop mention of uint from the ipldfree implementation.  So far all spec
discussions have tended towards mentioning "integers" as a type, and
definitely not "unsigned integers" or "positive-only integers" as a
distinct kind in the core Data Model.
Signed-off-by: Eric Myhre <hash@exultant.us>

That subject has more to do with SerializableNode.  Which also might
not be the name we end up with for that.  ID'able Node?  LinkableNode?
Signed-off-by: Eric Myhre <hash@exultant.us>

The very first draft tried to get away with *one* "Kind" enumeration,
but that quickly became odd and shakey; *two* separate "Kind" enums
(one for the Data Model, one for the lower level Representation;
working terms, and mine) fits a lot better.  The latter is what we're
committing here.

Also of interest here is a proposal for a distinction between whether
fields are *required* vs *nullable*.  I'm not sure this has been done
before in any of the other systems I've examined so far; it's a concept
I think we'll want for dealing with the subtle distinction between
whether some piece of data *matches* our schema vs whether it's *valid*
within our schema.  But it's quite hypothetical; it's possible this
whole concept of "matching" will turn out a lot more complex than that.

There's a tossed out syntax for a schema DSL in a comment.  This is
utterly unscrutinized and should not be taken too seriously yet.

The example code at the bottom declaring some type system is code that
*could* be used, but is mostly for demonstration and early dev
purposes: in the long run, we *do* want to come up with a DSL, and all
the relevant grammers, parsers, and so on for using that as an
implementation-agnostic source of truth.  At that (far future) point,
this kind of code would be used internally to represent what's been
parsed out of the DSL; but users shouldn't really be writing it.
(That's a long-winded way of saying "yes, some parts of that code are
extremely not DRY and would be error prone if written manually"; and
indeed, they would, and thus the point is not to.)
Signed-off-by: Eric Myhre <hash@exultant.us>

Finally got the bind and free impls on the same page.

Surprisingly, the bind node can work without being a ptr itself.
I'm not sure if that'll last, but let's try to roll with it.
If we *can* keep that property, it might reduce GC impact in a pretty
significant way.

Added a 'fluent' package.  It's just a twinkle in my eye so far, but it
might represent the nicest way through all the discussed issues.
Nodes shouldn't have to be panicful; and users shouldn't have to do all
error handling manually either.  A package full of fluent interfaces
that know what's going on seems to be the only way to get both.
But we'll see how this shakes out.  Maybe typeful traversers will
make the whole thing end up more coreward than being relegated to a
helper package.  I have no idea.
Signed-off-by: Eric Myhre <hash@exultant.us>

And add a bunch of type-specific ones for helpfulness.

But immediately re-reviewing this further.  What we've done here is
handle leafs very oddly, because it would seem to avoid unnecessary
wrapping on the leaf nodes.  But the result seems to be much uglier
code than it needs to be, and implies that we've got all sorts of
additional special cases required to either handle or reject binds
of primitive-kind fields.  And that seems... maybe not a great trade.
Signed-off-by: Eric Myhre <hash@exultant.us>

Note: I am not very pleased with how many deps this core and simple
data type has.  These are basically all for the hashing, and very, very
little actual hashing code at that, and I really wish the rule about
"a little vendoring is better than a little dependency" would be
applied here -- pulling in x/crypto and x/sys so that I can use a
type that's *a wrapper around string* is thoroughly ridiculous.
Signed-off-by: Eric Myhre <hash@exultant.us>

Some comments on the interface as well.  Almost certain that traversal
should be pulled off the node interface itself.
Signed-off-by: Eric Myhre <hash@exultant.us>

This has been sitting on my hard drive quite a while already, so I
decided to clean it up juuuust enough that it compiles so I can push
it.  But as you can see, there's a LOT of TODOs dangling here.

I'm not at all still sure those comments about "root node" in the
interface are going to stand up to scrutiny over time.  There's
definitely going to be some distinction between "Node we're traversing"
and "Node we are putting in a serialized block of bytes and can
think it's reasonable to address by hash", but figuring out the best
ergonomics of that is probably going to take a while.

I'm going to start on another simpler just-backed-by-maps-n-stuff Node
impl before going deeper with this one, because I suspect it'll make
tests easier to write, which will then make this whole pile of
reflection easier to test down the road as well.
Signed-off-by: Eric Myhre <hash@exultant.us>

Hopefully if we come up with a good way to flip one node impl into another,
we'll be able to come up with an easy interface for "this is referencing
an in-memory structure type of mine, but hashed as CBOR native standard".

We'll see if this is a good idea or not.  It's either a newer, cleaner,
lessons-learned set of libraries for using IPLD; or, it'll turn into a
"lesson-learned" itself and we'll fold the learnings back into the
existing libraries.  Time will tell!