GitLab

Moving from bindnode, and adding a Type method for consistency with
TypedNode.

Fixes #181.

It covers AssignKind, AssignNode, and AsKind for every combination of
assembler kind and method.

We also verify that a constructed scalar node behaves the same with
AsKind when using its representation, like the old test.

There's effectively a triple loop as a test table, so the subtest name
has up to three components separated by dashes, such as:

	TestSchema/Scalars/Bytes-AssignNode-String

We also use this test as a demo of quicktest instead of go-wish.

Finally, adapt bindnode to pass these tests just like codegen. This was
mainly a bunch of TODOs in the relevant methods.

Along with a generic Engine interface, so that they can be reused for
other ipld.Node implementations, such as bindnode.

node/bindnode will start using these in a follow-up commit, since this
one is large enough as is.

Tested that all three forms of testing schema/gen/go still work:

	go test

	CGO_ENABLED=0 go test

	go test -tags=skipgenbehavtests

The funciton is carefully documented via godoc, so I'm not going to
attempt to document it here again. But as a high-level summary, it's
like a reflect.DeepEqual applied to the ipld.Node interface rather than
reflect.Value.

The only other two noteworthy details are that errors are treated as
panics, and Links are compared directly via ==.

Finally, we add table-driven tests to ensure all edge cases work.

Fixes #173.

Now that we buffer the output, using go/format is trivial.

This makes the default behavior better, and means not having to use an
extra gofmt go:generate step everywhere.

And re-generate all code in this module.

This gets us to a point where go-codec-dagpb has zero vet warnings, for
example. schema/dmt still has a few warnings, but those are trickier to
fix, so will require another PR.

With this change, running 'go generate ./...' on the entire module while
running gopls on one of its files drops gopls's CPU spinning from ~25s
to well under a second. They should improve that anyway, but there's no
reason for the tens of thousands of tiny FS writes on our end either.

The time to run 'go generate ./...' itself is largely unaffected; it
goes from ~1.2s to ~1.1s, judging by a handful of runs.

If we know that a schema type can be represented in Go with a small
amount of bytes, using a pointer to store its "maybe" is rarely a good
idea. For example, an optional string only weighs twice as much as a
pointer, so a pointer adds overhead and will barely ever save any
memory.

Add a function to work out the byte size of a schema.TypeKind, relying
on reflection and the basicnode package. Debug prints are also present
if one wants to double-check the numbers. As of today, they are:

	sizeOf(small): 32 (4x pointer size)
	sizeOf(Bool): 1
	sizeOf(Int): 8
	sizeOf(Float): 8
	sizeOf(String): 16
	sizeOf(Bytes): 24
	sizeOf(List): 24
	sizeOf(Map): 32
	sizeOf(Link): 16

Below is the result on go-merkledag's BenchmarkRoundtrip after
re-generating go-codec-dagpb with this change. Note that the dag-pb
schema contains multiple optional fields, such as strings.

	name         old time/op    new time/op    delta
	Roundtrip-8    4.24µs ± 3%    3.78µs ± 0%  -10.87%  (p=0.004 n=6+5)

	name         old alloc/op   new alloc/op   delta
	Roundtrip-8    6.38kB ± 0%    6.24kB ± 0%   -2.26%  (p=0.002 n=6+6)

	name         old allocs/op  new allocs/op  delta
	Roundtrip-8       103 ± 0%        61 ± 0%  -40.78%  (p=0.002 n=6+6)

Schema typekinds which don't directly map to basicnode prototypes, such
as structs and unions, are left as a TODO for now.

I did not do any measurements to arrive at the magic number of 4x, which
is documented in the code. We might well increase it in the future, with
more careful benchmarking. For now, it seems like a conservative starting
point that should cover all basic types.

Finally, re-generate within this repo.

This significantly reworks how linking is handled.

All of the significant operations involved in storing and loading
data are extracted into their own separate features, and the LinkSystem
just composes them.  The big advantage of this is we can now add as
many helper methods to the LinkSystem construct as we want -- whereas
previously, adding methods to the Link interface was a difficult
thing to do, because that interface shows up in a lot of places.

Link is now *just* treated as a data holder -- it doesn't need logic
attached to it directly.  This is much cleaner.

The way we interact with the CID libraries is also different.
We're doing multihash registries ourselves, and breaking our direct
use of the go-multihash library.  The big upside is we're now using
the familiar and standard hash.Hash interface from the golang stdlib.
(And as a bonus, that actually works streamingly; go-mulithash didn't.)
However, this also implies a really big change for downstream users:
we're no longer baking as many hashes into the new multihash registry
by default.

I'm about to call it quits on this.  I'm not sure exactly where this
got off the rails, but I'm not happy about how its going, and
with this diff, I've reached enough "huh,hmm" moments that I think
it's going to end up being less work restarting on a cleaner approach
than it's going to be work finishing this, fixing all the bugs
resulting from the mess of maybeism, and then maintaining it.

Comments in the diff body show the exact moment of my exasperation
reaching a critical threshhold.

I'm really not happy with the golang typesystem today.

A more systematic review of this stack of diffs will follow in the
subsequent commit message.  It will be a merge-ignore commit.

This'll probably be a lot of grind to refactor, but otherwise isn't
a huge jump semantically.

I waffled on sprinting straight ahead to writing schema DMT as JSON,
and using those as the test fixture input.  On the one hand: it would
be nice to get that much closer to purely textual test fixtures.
On the other hand, it's really verbose, and I don't want to (long run,
we'll use schema DSL for this, and while we'll also save the DMT
fixtures, they should ideally probably mostly be generated and just
human-checked rather than human-penned);
and also, since we don't have implicits implemented correctly yet,
we'd need to update all that JSON again anyway once that feature is
complete... which pushes it overall to a net "nah".

Begin using it in some parts of testing of schema parse&compile.
(I don't think I'll try to push all usage of go-wish over to quicktest
in one flurry of diffs right now.  But it might be future work.)

The breaking point is that asserting on lists of errors using go-wish
was starting to exhibit Weird behaviors for only *some* error types.
It's likely that this would be addressable using some kind of go-cmp
customization, but go-wish doesn't really expose that ability.
The quicktest library does.

Switching to quicktest also brings a bunch of other nice features
along with it, like stack traces and other forms of additional info.
The holistic feel of it is also pretty similar to go-wish (particularly
since https://github.com/frankban/quicktest/pull/77 ), so many things
should be easy to switch.  I suspect I might want some more checker
functions to assert on types... but those should be easy to add as
a third party, either as a go-cmp.Transformer or a qt.Checker (and
then we can work on upstreaming at leisure, if appropriate).

In this commit, I'm handling the error list situation using a
go-cmp.Transformer to stringify all the errors.  This means that the
error *types* aren't checked, which is definitely loses ditches some
information... but the upside is that its easy to use the tests to
check that the eventual string of the error message is human-readable.
In this API, I think that readability is the higher priority.

Use a "__" pattern -- the same one already used in our schema codegen.

This is less ambiguous given we have some other names which already use
single underscores to demarcate other semantics (e.g. union membership
groupings).

Also things now ready left-to-right as is normal for golang (e.g., the
word "map" or "list" now comes first).

No logical changes.  Just naming.

Extracted a few last comment hunks that were useful and that's
the end of it.

The migrated tests only mostly pass (but, remember, we've been several
commits in a row already where we're making checkpoints to keep this
refactor managable, and CI is not full green across the board for
some of them).  I've updated some of the error text assertions,
but there's other breakage as well.  One is just other todos.
Another is... It may be time to switch test libraries.  What go-wish
(and transitively, go-cmp) is doing with `[]error` values is...
not helpful.

And I think this now ports all the rules we'd previously written
against the attempted schema2 (the wrapper style) API.
So I can now remove the rest of that in the next commit.

It will soon be time to start updating all the gengo stuff to
use this, including all of its tests.

Also, shift some error message text towards a more consistent phrasing.

Commit to the strategy of having the first flunked rule for a type
result in short-circuit skipping of subsequent rules.  It's simple,
and it's sufficient.

More rules are still to come.

Both are now accessible.  Name is not always present.

Get rid of casts that are unnecessary.

Constructors for anonymous types are still upcoming;
all the current constructors dupe the name into the reference field.
Planning to add distinct methods on the Compiler for anon types.

It details a variety of considered approaches.

Spoiler: I'm not actually super pleased with the one I'm currently
pursuing.  The amount of boilerplate I'm grinding out for this is
really, really no fun at all.  It's possibly that the reasoning leading
here is still sound.  It's just unpleasant.

Carving out hunks of the schema2 implementation of them (which still
hoped to use the dmt more directly) as I port them.

As comments in the diff state: I had a hope here that I could make
this relatively table-driven, which would increase legibility and
make it easier to port these checks to other implementations as well.
We'll... see how that goes; it's not easy to flatten.

schema: so much boilerplate for feeding information to the Compiler that I wrote another supplementary code generator.

(I'm getting very weary of golang.)

This new bit of codegen makes the compiler.go file fairly readable
again, though, so I'm satisfied with it.

The Compiler API is now complete enough that I can start repairing
other things to use it properly.  The schemadmt.Schema.Compile()
function and all of its helpers compile again now.  So does *most*
of the whole codegen system... with the notable exception of all
the hardcoded typesystem spawning which used the old placeholder
methods which have now been stricken.

TypeSystem now maintains order.  This allowed me to remove some
sort operations from the code generator.  This also means the next
time any existing codegen is re-run, the output file will shift
significantly.  However, it shouldn't do so again in the future.

As with parent commit: this is a checkpoint.  CI will not be passing.

This commit does not pass CI or even fully compile, and while I usually
try to avoid those, A) I need a checkpoint!, and B) I think this one is
interestingly illustrative, and I'll probably want to refer to this
diff and the one that will follow it in the future as part of
architecture design records (or even possibly experience reports about
golang syntax).

In this commit: we have three packages:

- schema: full of interfaces (and only interfaces)
- schema/compiler: creates values matching schema interfaces
- schema/dmt: contains codegen'd types that parse schema documents.

The dmt package feeds data to the compiler package, and the compiler
package emits values matching the schema interface.
This all works very nicely and avoids import cycles.

(Avoiding import cycles has been nontrivial, here, unfortunately.
The schema/schema2 package (which is still present in this commit,
but will be removed shortly -- I've scraped most of it over into
this new 'compiler' package already, just not a bunch of the validation
rules stuff, yet) was a dream of making this all work by just having
thin wrapper types around the dmt types.  This didn't fly...
because codegen'd nodes comply with `schema.TypedNode`, and complying
with `schema.TypedNode` means they have a function which references
`schema.Type`... and that means we really must depend on that
interface and the package it's in.  Ooof.)

The big downer with this state, and why things are currently
non-compiling at this checkpoint I've made here, is that we have to
replicate a *lot* of methods into single-use interfaces in the schema
package for this to work.  This belies the meaning of "interface".
The reason we'd do this -- the reason to split 'compiler' into its own
package -- is most because I wanted to keep all the constructor
mechanisms for schema values out of the direct path of the user's eye,
because most users shouldn't be using the compiler directly at all.

But... I'm shifting to thinking this attempt to segregate the compiler
details isn't worth it.  A whole separate package costs too much.
Most concretely, it would make it impossible to make the `schema.Type`
interface "closed" (e.g. by having an unexported method), and I think
at that point we would be straying quite far from desired semantics.

This means we no longer clutter the repository with lots of files, even
if they are git-ignored. It's always a bit of a red flag when you run
"go test ./..." and the result is a bunch of leftover files.

We still want to keep the files around, for the sake of Go's build
cache. And we still want their paths to be static between "go test"
runs. So put them in a static dir under os.TempDir.

This does mean that concurrent runs of these tests will likely not work
well. I don't imagine that's going to be a problem anytime soon, though.
If it really becomes a problem in the future, we could figure something
out like grabbing a file lock for the directory.

The idea behind using os.TempDir is that it will likely remain in place
between a number of "go test" runs within a hacking session, but it will
be eventually cleaned up by the system, such as when rebooting.

Note that we need to use globbing since one can't build "proper
packages" located outside a module. The only exception is building an
ad-hoc set of explicit Go files. While at it, use filepath.Join, to be
nice.

In particular, use keys for ipld error structs. These have one field, so
the changes are pretty simple.

Reduces 'go vet ./...' from 2647 lines of output to 2365.

Updates #102.

Practically every subtest ends up at 7 or so levels of names, like:

	TestMapsContainingMaybe/maybe-using-ptr/generate/compile/bhvtest/non-nullable/typed-create

However, note that the "generate" and "compile" levels are always there,
so their presence just adds verbosity in the output and makes the
developer's life more difficult.

Extremely nested sub-tests are already rare, so at least we can just
keep the components that add useful information in the output.

"bhvtest" is also pretty redundant, but that one actually matters - its
subtest can be skipped depending on build tags.

In particular, this removes ~50 out of the 2.7k warnings in 'go vet
./...' in this repository. Mainly, the "unreachable code" ones.

This was caused by edge cases in some of the generated code which caused
an unconditional return or panic statement to be followed by other code.
Fix all of them with a bit more template logic.

Some of the Next methods go a bit further. If they serve no purpose as
the switch has no cases to be matched, just unconditionally return an
error. In the future we can perhaps reuse a single function for that.

Finally, I was having a hard time actually following the logic in
kindedUnionNodeAssemblerMethodTemplateMunge, so I've indented the code a
bit to follow the template logic and scoping.

These changes move us towards pleasing vet, which is nice, but also make
the code waste a bit less space.

target of opporunity DRY improvement: use more shared templates for structs with stringjoin representations.

(Encountered while working on support unions with stringprefix representations.)