Merge branch 'master' into linksystem

Resolves conflicts in go.sum.
(They were conflicts in a textual sense only, but semantically trivial;
application of `go mod tidy` was sufficient.)

README.md

@@ -53,29 +53,47 @@ For overall behaviors and specifications, refer to the specs repo:
 ### distinctions from go-ipld-interface&go-ipld-cbor
 
 This library ("go ipld prime") is the current head of development for golang IPLD,
-but several other libraries exist which are widely deployed.
+and we recommend new developments in golang be done using this library as the basis.
 
-This library is a clean take on the IPLD interfaces and addresses several design decisions very differently than existing libraries:
+However, several other libraries exist in golang for working with IPLD data.
+Most of these predate go-ipld-prime and no longer receive active development,
+but since they do support a lot of other software, you may continue to seem them around for a while.
+go-ipld-prime is generally **serially compatible** with these -- just like it is with IPLD libraries in other languages.
 
-- The Node interfaces are minimal (and match cleanly to the IPLD Data Model);
+In terms of programmatic API and features, go-ipld-prime is a clean take on the IPLD interfaces,
+and chose to address several design decisions very differently than older generation of libraries:
+
+- **The Node interfaces map cleanly to the IPLD Data Model**;
 - Many features known to be legacy are dropped;
-- The Link implementations are purely CIDs;
+- The Link implementations are purely CIDs (no "name" nor "size" properties);
 - The Path implementations are provided in the same box;
 - The JSON and CBOR implementations are provided in the same box;
-- And several odd dependencies on blockstore and other interfaces from the rest of the IPFS ecosystem are removed.
+- Several odd dependencies on blockstore and other interfaces that were closely coupled with IPFS are replaced by simpler, less-coupled interfaces;
+- New features like IPLD Selectors are only available from go-ipld-prime;
+- New features like ADLs (Advanced Data Layouts), which provide features like transparent sharding and indexing for large data, are only available from go-ipld-prime;
+- Declarative transformations can be applied to IPLD data (defined in terms of the IPLD Data Model) using go-ipld-prime;
+- and many other small refinements.
+
+In particular, the clean and direct mapping of "Node" to concepts in the IPLD Data Model
+ensures a much more consistent set of rules when working with go-ipld-prime data, regardless of which codecs are involved.
+(Codec-specific embellishments and edge-cases were common in the previous generation of libraries.)
+This clarity is also what provides the basis for features like Selectors, ADLs, and operations such as declarative transformations.
 
 Many of these changes had been discussed for the other IPLD codebases as well,
 but we chose clean break v2 as a more viable project-management path.
-Both the existing IPLD libraries and go-ipld-prime can co-exist on the same import path, and refer to the same kinds of serial data.
+Both go-ipld-prime and these legacy libraries can co-exist on the same import path, and both refer to the same kinds of serial data.
 Projects wishing to migrate can do so smoothly and at their leisure.
 
-There is no explicit deprecation timeline for the earlier golang IPLD libraries,
-but you should expect new features *here*, rather than in those libraries.
+We now consider many of the earlier golang IPLD libraries to be defacto deprecated,
+and you should expect new features *here*, rather than in those libraries.
+(Those libraries still won't be going away anytime soon, but we really don't recomend new construction on them.)
+
+### unixfsv1
 
 Be advised that faculties for dealing with unixfsv1 data are still limited.
-You can find some tools in the [go-ipld-prime-proto](https://github.com/ipld/go-ipld-prime-proto/) repo,
-but be sure to read the caveats and limitations in that project's readme.
-We're happy to accept major PRs on this topic, though, if you who is reading this wants to fix this faster than wait for us :)
+You can find some tools for dealing with dag-pb (the underlying codec) in the [ipld/go-codec-dagpb](https://github.com/ipld/go-codec-dagpb) repo,
+and there are also some tools retrofitting some of unixfsv1's other features to be perceivable using an ADL in the [ipfs/go-unixfsnode](https://github.com/ipfs/go-unixfsnode) repo...
+however, a "some assembly required" advisory may still be in effect; check the readmes in those repos for details on what they support.
 
 
 
@@ -86,7 +104,7 @@ The go-ipld-prime library is already usable.  We are also still in development,
 
 A changelog can be found at [CHANGELOG.md](CHANGELOG.md).
 
-Using a commit hash when depending on this library is advisable (as it is with any other).
+Using a commit hash to pin versions precisely when depending on this library is advisable (as it is with any other).
 
 We may sometimes tag releases, but it's just as acceptable to track commits on master without the indirection.
 
@@ -109,3 +127,30 @@ None of this is to say we'll go breaking things willy-nilly for fun; but it *is*
 
 - Staying close to master is always better than not staying close to master;
 - and trust your compiler and your tests rather than tea-leaf patterns in a tag string.
+
+
+### Version Names
+
+When a tag is made, version number steps in go-ipld-prime advance as follows:
+
+1. the number bumps when the lead maintainer says it does.
+2. even numbers should be easy upgrades; odd numbers may change things.
+3. the version will start with `v0.` until further notice.
+
+[This is WarpVer](https://gist.github.com/warpfork/98d2f4060c68a565e8ad18ea4814c25f).
+
+These version numbers are provided as hints about what to expect,
+but ultimately, you should always invoke your compiler and your tests to tell you about compatibility.
+
+
+### Updating
+
+**Read the [CHANGELOG](CHANGELOG.md).**
+
+Really, read it.  We put exact migration instructions in there, as much as possible.  Even outright scripts, when feasible.
+
+An even-number release tag is usually made very shortly before an odd number tag,
+so if you're cautious about absorbing changes, you should update to the even number first,
+run all your tests, and *then* upgrade to the odd number.
+Usually the step to the even number should go off without a hitch, but if you *do* get problems from advancing to an even number tag,
+A) you can be pretty sure it's a bug, and B) you didn't have to edit a bunch of code before finding that out.

codec/raw/codec.go

+// Package raw implements IPLD's raw codec, which simply writes and reads a Node
+// which can be represented as bytes.
+//
+// The codec can be used with any node which supports AsBytes and AssignBytes.
+// In general, it only makes sense to use this codec on a plain "bytes" node
+// such as github.com/ipld/go-ipld-prime/node/basic.Prototype.Bytes.
+package raw
+
+import (
+	"fmt"
+	"io"
+	"io/ioutil"
+
+	ipld "github.com/ipld/go-ipld-prime"
+	cidlink "github.com/ipld/go-ipld-prime/linking/cid"
+)
+
+// TODO(mvdan): make go-ipld-prime use go-multicodec soon
+const rawMulticodec = 0x55
+
+func init() {
+	cidlink.RegisterMulticodecDecoder(rawMulticodec, Decode)
+	cidlink.RegisterMulticodecEncoder(rawMulticodec, Encode)
+}
+
+// Decode implements decoding of a node with the raw codec.
+//
+// Note that if r has a Bytes method, such as is the case with *bytes.Buffer, we
+// will use those bytes directly to save having to allocate and copy them. The
+// Node interface is defined as immutable, so it is assumed that its bytes won't
+// be modified in-place. Similarly, we assume that the incoming buffer's bytes
+// won't get modified in-place later.
+//
+// To disable the shortcut above, hide the Bytes method by wrapping the buffer
+// with an io.Reader:
+//
+//     Decode([...], struct{io.Reader}{buf})
+func Decode(am ipld.NodeAssembler, r io.Reader) error {
+	var data []byte
+	if buf, ok := r.(interface{ Bytes() []byte }); ok {
+		data = buf.Bytes()
+	} else {
+		var err error
+		data, err = ioutil.ReadAll(r)
+		if err != nil {
+			return fmt.Errorf("could not decode raw node: %v", err)
+		}
+	}
+	return am.AssignBytes(data)
+}
+
+// Encode implements encoding of a node with the raw codec.
+//
+// Note that Encode won't copy the node's bytes as returned by AsBytes, but the
+// call to Write will typically have to copy the bytes anyway.
+func Encode(node ipld.Node, w io.Writer) error {
+	data, err := node.AsBytes()
+	if err != nil {
+		return err
+	}
+	_, err = w.Write(data)
+	return err
+}

codec/raw/codec_test.go

+package raw
+
+import (
+	"bytes"
+	"context"
+	"fmt"
+	"io"
+	"testing"
+
+	qt "github.com/frankban/quicktest"
+	"github.com/ipfs/go-cid"
+	ipld "github.com/ipld/go-ipld-prime"
+	cidlink "github.com/ipld/go-ipld-prime/linking/cid"
+	basicnode "github.com/ipld/go-ipld-prime/node/basic"
+)
+
+var tests = []struct {
+	name string
+	data []byte
+}{
+	{"Empty", nil},
+	{"Plaintext", []byte("hello there")},
+	{"JSON", []byte(`{"foo": "bar"}`)},
+	{"NullBytes", []byte("\x00\x00")},
+}
+
+func TestRoundtrip(t *testing.T) {
+	t.Parallel()
+
+	for _, test := range tests {
+		t.Run(test.name, func(t *testing.T) {
+			nb := basicnode.Prototype.Bytes.NewBuilder()
+			r := bytes.NewBuffer(test.data)
+
+			err := Decode(nb, r)
+			qt.Assert(t, err, qt.IsNil)
+			node := nb.Build()
+
+			buf := new(bytes.Buffer)
+			err = Encode(node, buf)
+			qt.Assert(t, err, qt.IsNil)
+
+			qt.Assert(t, buf.Bytes(), qt.DeepEquals, test.data)
+		})
+	}
+}
+
+func TestRoundtripCidlink(t *testing.T) {
+	t.Parallel()
+
+	lb := cidlink.LinkBuilder{Prefix: cid.Prefix{
+		Version:  1,
+		Codec:    rawMulticodec,
+		MhType:   0x17,
+		MhLength: 4,
+	}}
+	node := basicnode.NewBytes([]byte("hello there"))
+
+	buf := bytes.Buffer{}
+	lnk, err := lb.Build(context.Background(), ipld.LinkContext{}, node,
+		func(ipld.LinkContext) (io.Writer, ipld.StoreCommitter, error) {
+			return &buf, func(lnk ipld.Link) error { return nil }, nil
+		},
+	)
+	qt.Assert(t, err, qt.IsNil)
+
+	nb := basicnode.Prototype__Any{}.NewBuilder()
+	err = lnk.Load(context.Background(), ipld.LinkContext{}, nb,
+		func(lnk ipld.Link, _ ipld.LinkContext) (io.Reader, error) {
+			return bytes.NewReader(buf.Bytes()), nil
+		},
+	)
+	qt.Assert(t, err, qt.IsNil)
+	qt.Assert(t, nb.Build(), qt.DeepEquals, node)
+}
+
+// mustOnlyUseRead only exposes Read, hiding Bytes.
+type mustOnlyUseRead struct {
+	buf *bytes.Buffer
+}
+
+func (r mustOnlyUseRead) Read(p []byte) (int, error) {
+	return r.buf.Read(p)
+}
+
+// mustNotUseRead exposes Bytes and makes Read always error.
+type mustNotUseRead struct {
+	buf *bytes.Buffer
+}
+
+func (r mustNotUseRead) Read(p []byte) (int, error) {
+	return 0, fmt.Errorf("must not call Read")
+}
+
+func (r mustNotUseRead) Bytes() []byte {
+	return r.buf.Bytes()
+}
+
+func TestDecodeBuffer(t *testing.T) {
+	t.Parallel()
+
+	var err error
+	buf := bytes.NewBuffer([]byte("hello there"))
+
+	err = Decode(
+		basicnode.Prototype.Bytes.NewBuilder(),
+		mustOnlyUseRead{buf},
+	)
+	qt.Assert(t, err, qt.IsNil)
+
+	err = Decode(
+		basicnode.Prototype.Bytes.NewBuilder(),
+		mustNotUseRead{buf},
+	)
+	qt.Assert(t, err, qt.IsNil)
+}

go.mod

@@ -3,6 +3,7 @@ module github.com/ipld/go-ipld-prime
 go 1.14
 
 require (
+	github.com/frankban/quicktest v1.11.3
 	github.com/ipfs/go-cid v0.0.4
 	github.com/multiformats/go-multihash v0.0.15
 	github.com/polydawn/refmt v0.0.0-20190807091052-3d65705ee9f1

go.sum

+github.com/frankban/quicktest v1.11.3 h1:8sXhOn0uLys67V8EsXLc6eszDs8VXWxL3iRvebPhedY=
+github.com/frankban/quicktest v1.11.3/go.mod h1:wRf/ReqHper53s+kmmSZizM8NamnL3IM0I9ntUbOk+k=
+github.com/google/go-cmp v0.5.4 h1:L8R9j+yAqZuZjsqh/z+F1NCffTKKLShY6zXTItVIZ8M=
+github.com/google/go-cmp v0.5.4/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
 github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1 h1:EGx4pi6eqNxGaHF6qqu48+N2wcFQ5qg5FXgOdqsJ5d8=
 github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
 github.com/ipfs/go-cid v0.0.4 h1:UlfXKrZx1DjZoBhQHmNHLC1fK1dUJDN20Y28A7s+gJ8=
@@ -6,6 +10,11 @@ github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7
 github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
 github.com/klauspost/cpuid/v2 v2.0.4 h1:g0I61F2K2DjRHz1cnxlkNSBIaePVoJIjjnHui8QHbiw=
 github.com/klauspost/cpuid/v2 v2.0.4/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
+github.com/kr/pretty v0.2.1 h1:Fmg33tUaq4/8ym9TJN1x7sLJnHVwhP33CNkpYV/7rwI=
+github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
+github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
+github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
+github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
 github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1 h1:lYpkrQH5ajf0OXOcUbGjvZxxijuBwbbmlSxLiuofa+g=
 github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1/go.mod h1:pD8RvIylQ358TN4wwqatJ8rNavkEINozVn9DtGI3dfQ=
 github.com/minio/sha256-simd v0.1.1-0.20190913151208-6de447530771/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM=
@@ -49,3 +58,5 @@ golang.org/x/sys v0.0.0-20210309074719-68d13333faf2/go.mod h1:h1NjWce9XRLGQEsW7w
 golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
+golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=