GitLab

Lots of individual things:

- removed the "Begin*" functions; no need to expose that kind of raw
  operation when the callback forms are zero-cost.

- renamed functions for consistent "Build" vs "Assemble".

- added "Assign*" functions for all scalar kinds, which reduces the
  usage of "AbsorbError" (but also, left AbsorbError in).

- renamed the ListEntry/MapEntry functions to also have "Assemble*"
  forms (still callback style).

- while also adding Assign{Map|List}Entry{Kind} functions (lets you
  get rid of another callback whenever the value is a scalar).

- added Assign{|MapEntry|ListEntry} functions, which shell out to
  fluent.Reflect for even more convenience (at the cost of performance).

- moved higher level functions like CopyRange to a separate file.

- new benchmark, for the terser form of working with scalars.
  (It's also evidently slightly faster, because fewer small function
  calls.  Slightly surprising considering how much inlining we might
  expect, but, huh.  Alright, surprise bonus; acceptable.)

- example function updated to use the terser form.

With these terseness improvements to handling of scalars, the overall
SLOC count for using the quip system is now exactly on par with fluent.

Varations on map key arguments (which could be PathSegment or even
Node, in addition to string) still aren't made available this.
Perhaps that's just okay.  If you're really up some sort of creek
where you need that, you can still just use the
MapAssembler.AssembleKey system directly, which can do everything.

If there's any furhter action to take after the callback, it's
necessary to recheck the error slot before taking that action;
and of course we definitely don't want to overwrite the error if one
had been set during the callback.

I wonder if it would ever be useful to have a variant of these
functions which *does* attempt to call `Finish`, etc, and thus still
build a partial tree at the end even if it stopped on error midway.
(Right now, if something stops midway, the final `Build` call will
panic, and tell you you haven't finished all the assemblers.)
It would be a bit more complicated though: we'd potentially need to
accumulate more than one error.  And in practice, when working on
schema'd data, it would often still result in invalid results
(anything other than optional fields, or type-level maps and lists,
will fail some other logical rule if not fully filled in), which
makes me wonder how often this would be useful.

This reverts commit 6e6625bd.

Discussed at
https://github.com/ipld/go-ipld-prime/pull/126#issuecomment-753003441

Long story short, the motivations of this rename are good,
but the new name also carries some connotations we're really not sure
about, and so we're going to undo this for now, and continue to think
about it in the future.

Implement traversal.FocusedTransform.

And a few new accessors for Path that are helpful and reasonable.

all: rename schema.Kind to TypeKind, ipld.ReprKind to Kind

As discussed on the issue thread, ipld.Kind and schema.TypeKind are more
intuitive, closer to the spec wording, and just generally better in the
long run.

The changes are almost entirely automated via the commands below. Very
minor changes were needed in some of the generators, and then gofmt.

	sed -ri 's/\<Kind\(\)/TypeKind()/g' **/*.go
	git checkout fluent # since it uses reflect.Value.Kind

	sed -ri 's/\<Kind_/TypeKind_/g' **/*.go
	sed -i 's/\<Kind\>/TypeKind/g' **/*.go
	sed -i 's/ReprKind/Kind/g' **/*.go

Plus manually undoing a few renames, as per Eric's review.

Fixes #94.

This should be more intuitive to Go programmers, since assignments are
generally trivial operations, but conversions imply that extra work
might be needed to adapt the value to fit in the recipient.

The entire change is just:

	sed -ri 's/AssignNode/ConvertFrom/g' **/*.go

Downstream users can very likely use the same line to fix their function
declarations and calls.

Fixes #95.

We only supported representing Int nodes as Go's "int" builtin type.
This is fine on 64-bit, but on 32-bit, it limited those node values to
just 32 bits. This is a problem in practice, because it's reasonable to
want more than 32 bits for integers.

Moreover, this meant that IPLD would change behavior if built for a
32-bit platform; it would not be able to decode large integers, for
example, when in fact that was just a software limitation that 64-bit
builds did not have.

To fix this problem, consistently use int64 for AsInt and AssignInt.

A lot more functions are part of this rewrite as well; mainly, those
revolving around collections and iterating. Some might never need more
than 32 bits in practice, but consistency and portability is preferred.
Moreover, many are interfaces, and we want IPLD interfaces to be
flexible, which will be important for ADLs.

Below are some GNU sed lines which can be used to quickly update
function signatures to use int64:

	sed -ri 's/(func.* AsInt.*)\<int\>/\1int64/g' **/*.go
	sed -ri 's/(func.* AssignInt.*)\<int\>/\1int64/g' **/*.go
	sed -ri 's/(func.* Length.*)\<int\>/\1int64/g' **/*.go
	sed -ri 's/(func.* LookupByIndex.*)\<int\>/\1int64/g' **/*.go
	sed -ri 's/(func.* Next.*)\<int\>/\1int64/g' **/*.go
	sed -ri 's/(func.* ValuePrototype.*)\<int\>/\1int64/g' **/*.go

Note that the function bodies, as well as the code that calls said
functions, may need to be manually updated with the integer type change.
That cannot be automated, because it's possible that an automated fix
would silently introduce potential overflows not being handled.

Some TODOs and FIXMEs for overflow checks are removed, since we remove
some now unnecessary int64->int conversions. On the other hand, the
older codecs based on refmt need to gain some overflow check TODOs,
since refmt uses ints. That is okay for now, since we'll phase out refmt
pretty soon.

While at it, update codectools to use int64 for token Length fields, so
that it properly supports full IPLD integers without machine-dependent
behavior and overflow checks. The budget integer is also updated to be
int64, since the lengths it uses are now int64.

Note that this refactor needed changes to the Go code generator as well
as some of the tests, for the purpose of updating all the code.

Finally, note that the code-generated iterator structs do not use int64
fields internally, even though they must return int64 numbers to
implement the interface. This is because they use the numeric fields to
count up to a small finite amount (such as the number of fields in a Go
struct), or up to the length of a map/slice. Neither of them can ever
outgrow "int".

Fixes #124.

clean up node/gendemo regeneration

Do regeneratation.  Some diff churn: the codegen doesn't yet jive
smoothly with a recent change made in service to go vet.
We'll want to straighten that out soon, I guess.

Schema types rebased to use codegen types for the data

Cannot quite wire that up yet because of some other still incomplete features.

Since https://github.com/ipld/go-ipld-prime/pull/121, presence of
fields is actually checked... but that code also doesn't understand
implicit fields yet, which makes us need a lot of filler.

Also the lack of the "members" field for unions?  That was just plain
wrong.  Good think we're catching things like that now.

filenames change due to https://github.com/ipld/go-ipld-prime/pull/105 .

gofmt also applied for the first time.

from here on out: `go generate` should just cause these files to be
automagically updated and formatted.

Move the existing setup from the schema-schema "demo" dir to here;
and rig it up with go generate conventions that I'm hoisting back from
mvdan's https://github.com/ipld/go-ipld-adl-hamt/blob/master/gen.go .

Move the parse tests with it.

draft of schema types using codegen for data model, with a package for the fully validated data which is implemented by retaining and accessing into the raw data.

codegen: assembler for struct with map representation validates all non-optional fields are present

codegen: assembler for struct with map representation now validates all non-optional fields are present.

This continues what https://github.com/ipld/go-ipld-prime/pull/111/ did
and adds the same logic to the map representation.  The actual state
tracking works the same way (and was mostly already there).

Rearranged the tests slightly.

Made error messages include both field name and serial key when they
differ due to a rename directive.  (It's possible this error would get
nicer if it used a list of StructField instead of just strings, but it
would also get more complicated.  Maybe revisit later.)

(Trying to call Build on an assembler that previously errored is very
likely to panic, so the fluent.Build function should return before
trying to do that.)

Reduces the output of 'go vet ./...' from 374 lines to 96. Many warnings
remain, but I have lost my patience for today.

Most of the changes below were automated, especially the single-line
mixins expressions. Unfortunately, many of the Traits structs required
manual copy-pasting.

This is the mixins package, so "Mixin" in filenames is redundant.

The files in this codegen demo correspond to an older version of the Go
code generator. That's not terrible in itself, but it did make repeated
uses of 'go test' fail:

	$ go test
	testing: warning: no tests to run
	PASS
	ok  	github.com/ipld/go-ipld-prime/node/gendemo	0.112s
	$ go test
	# github.com/ipld/go-ipld-prime/node/gendemo [github.com/ipld/go-ipld-prime/node/gendemo.test]
	./minima.go:10:2: midvalue redeclared in this block
		previous declaration at ./ipldsch_minima.go:13:27
	[...]

The original idea of this branch was to explore some reusable
components for codecs; maybe to actually get a useful
prettyprinter; and... actually turned a lot into being a bit of
practical discovery about string encoding and escaping.

Some of those goals were partially accomplished, but in general,
this seems better to chalk up as a learning experience.
https://github.com/ipld/go-ipld-prime/pull/89#issuecomment-703327909
already does a good job of discussing why, and what as learned.

A lot of the reusable codec components stuff has also now
shaken out, just... in other PRs that were written after the learnings
here.  Namely, https://github.com/ipld/go-ipld-prime/pull/101/
was able to introduce some tree transformers; and then
https://github.com/ipld/go-ipld-prime/pull/112 demonstrates how those
can compose into a complete codec end to end.
There's still work to go on these, too, but they seem to have already
grabbed the concept of reusable parts I was hoping for here and gotten
farther with it, so.

These diffs are interesting enough I want to keep them referencable
in history.  But I'm merging them with the "-s ours" strategy, so that
the diffs don't actually land any impact on master.  These commits
are for reference only.

It still uses the codec/tools package, but it's very clearly not a
codec itself and shouldn't be grouped like one, despite the shared
common implementation details.

Renamed methods to "Stringify".

Stringify no longer returns errors; those errors only arise from
the writer erroring, and writing into an in-memory buffer can't error.

I think this, or a variant of it, may be reasonable to rig up as a
Stringer on the basicnode types, and recommend for other Node
implementations to use as their Stringer too.

It's a fairly verbose output: I'm mostly aiming to use it in examples.

Bytes in particular are fun: I decided to make them use the hex.Dump
format.  (Why not?)

I've put this in a codec sub-package, because in some ways it feels
like a codec -- it's something you can apply to any node, and it
streams data out to an io.Writer -- but it's also worth noting it's
not meant to be a multicodec or generally written with an intention
of use anywhere outside of debug printf sorts of uses.

The codectools package, although it only has this one user, is a
reaction to previous scenarios where I've wanted a quick debug method
and desperately wanted something that gives me reasonable quoted
strings... without reaching for a json package.  We'll see if it's
worth it over time; I'm betting yes, but not with infinite confidence.
(This particular string escaping function also has the benefit of
encoding even non-utf-8 strings without loss of information -- which
is noteworthy, because I've recently noticed JSON _does not_; yikes.)

In fact this became a docs change; it is often desirable to *not* use the cidlink.Link type as a pointer.

Trying to make CIDs only usable as a pointer would be nice from a
consistency perspective, but has other consequences.

It's easy to forget this (and I apparently just did), but...
We often use link types as map keys.  And this is Important.

That means trying to handle CIDs as pointers leads to nonsensical
results: pointers are technically valid as a golang map key, but
they don't "do the right thing" -- the equality check ends up operating
on the the pointer rather than on the data.  This is well-defined,
but generally useless for these types in context.

As the comments in the diff say: it's a fairly sizable footgun for
users to need to consider whether they expect the pointer form or
the bare form when inspecting what an `ipld.Link` interface contains:
so, let's just remove the choice.

There's technically no reason for the Link.Load method to need to be
attached to the pointer receiver other than removing this footgun.
From the other side, though, there's no reason *not* to make it
attached to the pointer receiver, because any time a value is assigned
to an interface type, it necessarily heap-escapes and becomes a pointer
anyway.  So, making it unconditional and forcing the pointer to be
clear in the user's hands seems best.

Codec revamp

I dearly wish this wasn't such a dark art.
But I really want these tests, too.