GitLab

As with parent commit: this is a checkpoint.  CI will not be passing.

This commit does not pass CI or even fully compile, and while I usually
try to avoid those, A) I need a checkpoint!, and B) I think this one is
interestingly illustrative, and I'll probably want to refer to this
diff and the one that will follow it in the future as part of
architecture design records (or even possibly experience reports about
golang syntax).

In this commit: we have three packages:

- schema: full of interfaces (and only interfaces)
- schema/compiler: creates values matching schema interfaces
- schema/dmt: contains codegen'd types that parse schema documents.

The dmt package feeds data to the compiler package, and the compiler
package emits values matching the schema interface.
This all works very nicely and avoids import cycles.

(Avoiding import cycles has been nontrivial, here, unfortunately.
The schema/schema2 package (which is still present in this commit,
but will be removed shortly -- I've scraped most of it over into
this new 'compiler' package already, just not a bunch of the validation
rules stuff, yet) was a dream of making this all work by just having
thin wrapper types around the dmt types.  This didn't fly...
because codegen'd nodes comply with `schema.TypedNode`, and complying
with `schema.TypedNode` means they have a function which references
`schema.Type`... and that means we really must depend on that
interface and the package it's in.  Ooof.)

The big downer with this state, and why things are currently
non-compiling at this checkpoint I've made here, is that we have to
replicate a *lot* of methods into single-use interfaces in the schema
package for this to work.  This belies the meaning of "interface".
The reason we'd do this -- the reason to split 'compiler' into its own
package -- is most because I wanted to keep all the constructor
mechanisms for schema values out of the direct path of the user's eye,
because most users shouldn't be using the compiler directly at all.

But... I'm shifting to thinking this attempt to segregate the compiler
details isn't worth it.  A whole separate package costs too much.
Most concretely, it would make it impossible to make the `schema.Type`
interface "closed" (e.g. by having an unexported method), and I think
at that point we would be straying quite far from desired semantics.

This means we no longer clutter the repository with lots of files, even
if they are git-ignored. It's always a bit of a red flag when you run
"go test ./..." and the result is a bunch of leftover files.

We still want to keep the files around, for the sake of Go's build
cache. And we still want their paths to be static between "go test"
runs. So put them in a static dir under os.TempDir.

This does mean that concurrent runs of these tests will likely not work
well. I don't imagine that's going to be a problem anytime soon, though.
If it really becomes a problem in the future, we could figure something
out like grabbing a file lock for the directory.

The idea behind using os.TempDir is that it will likely remain in place
between a number of "go test" runs within a hacking session, but it will
be eventually cleaned up by the system, such as when rebooting.

Note that we need to use globbing since one can't build "proper
packages" located outside a module. The only exception is building an
ad-hoc set of explicit Go files. While at it, use filepath.Join, to be
nice.

I started rewriting the "getting started" Go guide to use fluent/qp
instead of fluent, but I'm missing some of the helpers since the guide
uses links, among others.

This is largely copy-pasted, but there are just a handful of types and
it's barely a dozen lines per type. A generator is not worth it for 100
lines of code that will rarely ever need to change.

This is what I came up with, building on top of Eric's quip. I don't
want to waste too much time naming this, and I like two-letter package
names in place of dot-imports, so "qp" seems good enough for now. They
are the "strong" consonants when one says "Quick iPld".

First, move the benchmarks comparing all fluent packages to the root
fluent package, to keep things a bit more tidy.

Second, make all the benchmarks report their allocation stats, without
having to always remember to use the -benchmem flag.

Third, add a qp benchmark.

Fourth, notice a couple of potential bugs in the quip benchmarks, and
add TODOs for them.

Finally, add the qp API. It differs from quip in a few external ways:

1) No error pointers. Instead, it uses panics which are recovered at the
   top-level API layer. This reduces verbosity, removes the "forgot to
   handle an error" type of mistake, and does not affect performance
   thanks to the defers being statically allocated in the stack.

2) Supposed better composition. For example, one can use MapEntry along
   with Map to have a map inside another map. In contrast, quip requires
   either an extra layer of func literals, or extra API like
   AssignMapEntryString.

3) Thanks to the points above, the API is significantly smaller. Note
   that some helper APIs like Bool are missing, but even when added, qp
   should expose about half the API funcs taht quip does.

This is the first proof of concept. I'll probably finish adding the rest
of the API helpers when I find the first use case for qp.

Benchmark numbers, with perflock and benchstat on my i5-8350u laptop:

	name                              time/op
	Quip-8                            1.39µs ± 1%
	QuipWithoutScalarFuncs-8          1.42µs ± 2%
	Qp-8                              1.46µs ± 2%

	name                              alloc/op
	Quip-8                              912B ± 0%
	QuipWithoutScalarFuncs-8            912B ± 0%
	Qp-8                                912B ± 0%

	name                              allocs/op
	Quip-8                              18.0 ± 0%
	QuipWithoutScalarFuncs-8            18.0 ± 0%
	Qp-8                                18.0 ± 0%

In particular, use keys for ipld error structs. These have one field, so
the changes are pretty simple.

Reduces 'go vet ./...' from 2647 lines of output to 2365.

Updates #102.

Practically every subtest ends up at 7 or so levels of names, like:

	TestMapsContainingMaybe/maybe-using-ptr/generate/compile/bhvtest/non-nullable/typed-create

However, note that the "generate" and "compile" levels are always there,
so their presence just adds verbosity in the output and makes the
developer's life more difficult.

Extremely nested sub-tests are already rare, so at least we can just
keep the components that add useful information in the output.

"bhvtest" is also pretty redundant, but that one actually matters - its
subtest can be skipped depending on build tags.

%q is superior to the manually quoted "%s", since it will properly
escape the inner string when quoting. For example:

	"here goes a double quote: \" "

Even if this does not matter in practice, using %q is easier too.

In particular, this removes ~50 out of the 2.7k warnings in 'go vet
./...' in this repository. Mainly, the "unreachable code" ones.

This was caused by edge cases in some of the generated code which caused
an unconditional return or panic statement to be followed by other code.
Fix all of them with a bit more template logic.

Some of the Next methods go a bit further. If they serve no purpose as
the switch has no cases to be matched, just unconditionally return an
error. In the future we can perhaps reuse a single function for that.

Finally, I was having a hard time actually following the logic in
kindedUnionNodeAssemblerMethodTemplateMunge, so I've indented the code a
bit to follow the template logic and scoping.

These changes move us towards pleasing vet, which is nice, but also make
the code waste a bit less space.

Introduce 'quip' data building helpers.

Lots of individual things:

- removed the "Begin*" functions; no need to expose that kind of raw
  operation when the callback forms are zero-cost.

- renamed functions for consistent "Build" vs "Assemble".

- added "Assign*" functions for all scalar kinds, which reduces the
  usage of "AbsorbError" (but also, left AbsorbError in).

- renamed the ListEntry/MapEntry functions to also have "Assemble*"
  forms (still callback style).

- while also adding Assign{Map|List}Entry{Kind} functions (lets you
  get rid of another callback whenever the value is a scalar).

- added Assign{|MapEntry|ListEntry} functions, which shell out to
  fluent.Reflect for even more convenience (at the cost of performance).

- moved higher level functions like CopyRange to a separate file.

- new benchmark, for the terser form of working with scalars.
  (It's also evidently slightly faster, because fewer small function
  calls.  Slightly surprising considering how much inlining we might
  expect, but, huh.  Alright, surprise bonus; acceptable.)

- example function updated to use the terser form.

With these terseness improvements to handling of scalars, the overall
SLOC count for using the quip system is now exactly on par with fluent.

Varations on map key arguments (which could be PathSegment or even
Node, in addition to string) still aren't made available this.
Perhaps that's just okay.  If you're really up some sort of creek
where you need that, you can still just use the
MapAssembler.AssembleKey system directly, which can do everything.

gengo: support for unions with stringprefix representation.

If there's any furhter action to take after the callback, it's
necessary to recheck the error slot before taking that action;
and of course we definitely don't want to overwrite the error if one
had been set during the callback.

I wonder if it would ever be useful to have a variant of these
functions which *does* attempt to call `Finish`, etc, and thus still
build a partial tree at the end even if it stopped on error midway.
(Right now, if something stops midway, the final `Build` call will
panic, and tell you you haven't finished all the assemblers.)
It would be a bit more complicated though: we'd potentially need to
accumulate more than one error.  And in practice, when working on
schema'd data, it would often still result in invalid results
(anything other than optional fields, or type-level maps and lists,
will fail some other logical rule if not fully filled in), which
makes me wonder how often this would be useful.

target of opporunity DRY improvement: use more shared templates for structs with stringjoin representations.

(Encountered while working on support unions with stringprefix representations.)

Should not affect most user code; though these are technically
exported symbols, they're very unlikely to be used directly.

This reverts commit 6e6625bd.

Discussed at
https://github.com/ipld/go-ipld-prime/pull/126#issuecomment-753003441

Long story short, the motivations of this rename are good,
but the new name also carries some connotations we're really not sure
about, and so we're going to undo this for now, and continue to think
about it in the future.

Implement traversal.FocusedTransform.

And a few new accessors for Path that are helpful and reasonable.

all: rename schema.Kind to TypeKind, ipld.ReprKind to Kind

As discussed on the issue thread, ipld.Kind and schema.TypeKind are more
intuitive, closer to the spec wording, and just generally better in the
long run.

The changes are almost entirely automated via the commands below. Very
minor changes were needed in some of the generators, and then gofmt.

	sed -ri 's/\<Kind\(\)/TypeKind()/g' **/*.go
	git checkout fluent # since it uses reflect.Value.Kind

	sed -ri 's/\<Kind_/TypeKind_/g' **/*.go
	sed -i 's/\<Kind\>/TypeKind/g' **/*.go
	sed -i 's/ReprKind/Kind/g' **/*.go

Plus manually undoing a few renames, as per Eric's review.

Fixes #94.

This should be more intuitive to Go programmers, since assignments are
generally trivial operations, but conversions imply that extra work
might be needed to adapt the value to fit in the recipient.

The entire change is just:

	sed -ri 's/AssignNode/ConvertFrom/g' **/*.go

Downstream users can very likely use the same line to fix their function
declarations and calls.

Fixes #95.

We only supported representing Int nodes as Go's "int" builtin type.
This is fine on 64-bit, but on 32-bit, it limited those node values to
just 32 bits. This is a problem in practice, because it's reasonable to
want more than 32 bits for integers.

Moreover, this meant that IPLD would change behavior if built for a
32-bit platform; it would not be able to decode large integers, for
example, when in fact that was just a software limitation that 64-bit
builds did not have.

To fix this problem, consistently use int64 for AsInt and AssignInt.

A lot more functions are part of this rewrite as well; mainly, those
revolving around collections and iterating. Some might never need more
than 32 bits in practice, but consistency and portability is preferred.
Moreover, many are interfaces, and we want IPLD interfaces to be
flexible, which will be important for ADLs.

Below are some GNU sed lines which can be used to quickly update
function signatures to use int64:

	sed -ri 's/(func.* AsInt.*)\<int\>/\1int64/g' **/*.go
	sed -ri 's/(func.* AssignInt.*)\<int\>/\1int64/g' **/*.go
	sed -ri 's/(func.* Length.*)\<int\>/\1int64/g' **/*.go
	sed -ri 's/(func.* LookupByIndex.*)\<int\>/\1int64/g' **/*.go
	sed -ri 's/(func.* Next.*)\<int\>/\1int64/g' **/*.go
	sed -ri 's/(func.* ValuePrototype.*)\<int\>/\1int64/g' **/*.go

Note that the function bodies, as well as the code that calls said
functions, may need to be manually updated with the integer type change.
That cannot be automated, because it's possible that an automated fix
would silently introduce potential overflows not being handled.

Some TODOs and FIXMEs for overflow checks are removed, since we remove
some now unnecessary int64->int conversions. On the other hand, the
older codecs based on refmt need to gain some overflow check TODOs,
since refmt uses ints. That is okay for now, since we'll phase out refmt
pretty soon.

While at it, update codectools to use int64 for token Length fields, so
that it properly supports full IPLD integers without machine-dependent
behavior and overflow checks. The budget integer is also updated to be
int64, since the lengths it uses are now int64.

Note that this refactor needed changes to the Go code generator as well
as some of the tests, for the purpose of updating all the code.

Finally, note that the code-generated iterator structs do not use int64
fields internally, even though they must return int64 numbers to
implement the interface. This is because they use the numeric fields to
count up to a small finite amount (such as the number of fields in a Go
struct), or up to the length of a map/slice. Neither of them can ever
outgrow "int".

Fixes #124.

clean up node/gendemo regeneration

Do regeneratation.  Some diff churn: the codegen doesn't yet jive
smoothly with a recent change made in service to go vet.
We'll want to straighten that out soon, I guess.

Schema types rebased to use codegen types for the data

Cannot quite wire that up yet because of some other still incomplete features.

Since https://github.com/ipld/go-ipld-prime/pull/121, presence of
fields is actually checked... but that code also doesn't understand
implicit fields yet, which makes us need a lot of filler.

Also the lack of the "members" field for unions?  That was just plain
wrong.  Good think we're catching things like that now.

filenames change due to https://github.com/ipld/go-ipld-prime/pull/105 .

gofmt also applied for the first time.

from here on out: `go generate` should just cause these files to be
automagically updated and formatted.

Move the existing setup from the schema-schema "demo" dir to here;
and rig it up with go generate conventions that I'm hoisting back from
mvdan's https://github.com/ipld/go-ipld-adl-hamt/blob/master/gen.go .

Move the parse tests with it.

draft of schema types using codegen for data model, with a package for the fully validated data which is implemented by retaining and accessing into the raw data.