GitLab

We recently updated the spec to say that decoders "should" allow
decoding a PBNode in either order of its fields. That is because some
IPFS data out in the wild is already encoded in the non-canonical
format, which is what Protobuf uses as a default, too.

This change makes the decoder here comply with the spec change, adding a
test with the encoded block that caused this entire spec change in the
first place: go-ipfs/test/sharness/t0110-gateway-data/foofoo.block.

The change to the decoder is slightly subtle, because the decoder used
to leverage the knowledge that Links must come before Data to start the
Links list early, and finish it as soon as Data arrives.

Since that order is now unknown, we must have some extra logic to
support either order. We also need special code to make sure Links is
always set, just like before.

We also add a test to double check that Data between Links is rejected.

An io.Writer, by definition, will always copy bytes. That's fine in
general, and can't be worked around without breaking the writer's
contract.

However, the main use of go-codec-dagpb today is go-merkledag, which
simply uses the codec to encode a node into a buffer.

So, we had to create a new bytes.Buffer, write to it, and grab its
bytes. This is one extra allocation (the bytes.Buffer object itself),
plus copying the encoded bytes an extra time, since we must copy from
Encode's internal buffer to the bytes.Buffer.

Add a lower-level append-like AppendEncode that cuts the middle man,
removing both of those extra pieces of work.

For the sake of consistency, we add DecodeBytes to mirror the above on
the decode side. Decode already had a shortcut for Bytes, but this way
it's more evident what we're doing, and we also avoid allocating a
bytes.Buffer just to call Bytes on it.

Using these new APIs in go-merkledag shows nice results:

	name         old time/op    new time/op    delta
	Roundtrip-8    4.27µs ± 0%    4.07µs ± 0%  -4.50%  (p=0.004 n=5+6)

	name         old alloc/op   new alloc/op   delta
	Roundtrip-8    6.86kB ± 0%    6.38kB ± 0%  -6.99%  (p=0.002 n=6+6)

	name         old allocs/op  new allocs/op  delta
	Roundtrip-8       106 ± 0%       103 ± 0%  -2.83%  (p=0.002 n=6+6)

While at it, we formally deprecate Marshal and Unmarshal, since we're
starting to have lots of redundant API surface.

IPLD's codec helper reader has a relatively high cost, unfortunately. It
was the main contributor to a slowdown in go-merkledag when moving from
the old protobuf gogo-generated decoder to go-codec-dagpb.

Using a []byte also means we can reuse protobuf's well-optimized "wire
encoding" helpers, which gets us extra speed and allows removing some
code.

This should not matter in practice for the time being, as the only
go-codec-dagpb user is go-merkledag and it uses bytes.Buffer everywhere.

In the future it would be nice for go-codec-dagpb to be just as
efficient with a stream decoder, but right now I don't have the extra
week to get into that. Plus, if the core protobuf implementation works
on []byte, one can assume it's reasonable for us to do the same.

Using the new BenchmarkRoundtrip in go-merkledag with go-codec-dagpb, we
get a significant uplift in performance:

	name         old time/op    new time/op    delta
	Roundtrip-8    6.49µs ± 1%    5.34µs ± 1%  -17.74%  (p=0.002 n=6+6)

	name         old alloc/op   new alloc/op   delta
	Roundtrip-8    8.07kB ± 0%    7.50kB ± 0%   -7.04%  (p=0.002 n=6+6)

	name         old allocs/op  new allocs/op  delta
	Roundtrip-8       171 ± 0%       148 ± 0%  -13.45%  (p=0.002 n=6+6)

std's errors has had wrapping for years now.

Fixes: https://github.com/ipld/go-codec-dagpb/pull/3