Begin introduction of new linking interfaces.

This is going to be the start of a pretty hectic set of commits.

1. The LinkLoader, LinkBuilder, and LinkContext types currently in
  the traversal package are hereby doomed (and will be deleted by
  the time this branch is ready to land).

2. Cid itself will disappear from almost all remaining concrete uses.
  For example, traversal.TraversalProgress.LastBlock will use Link
  instead of Cid.  (You'll have to cast back to Cid if that detail
  is important to your application!)

3. The 'repose' package is pretty much getting nuked.

Instead: we have these new interfaces.  Link will be abstract.

We'll add a linking package with a subpackage containing implementation
with Cids.  Our encoding systems will then also live there: this makes
sense since multicodec is definitely a detail associated with Cids.
We'll also have dag-cbor and dag-json specific encodings in subpackages
associated with this whole thing: those will be able to read and reify
Link instances during their handling of serial data.  This too is
parsimonious and correct (e.g. dag-json parsing is technically distinct
from regular json parsing, even if it's experientially close).

And if all goes according to plan, we will -- shockingly -- be able to
use almost the entirety of the IPLD system... *without* forming an
explicit import dependency on the Cid packages *until* we directly use
any link loading features based on Cids.  That'll be neat.

All these changes will be staged across a series of commits because
the total diff will definitely be something fierce.
Signed-off-by: Eric Myhre <hash@exultant.us>

linking.go

+package ipld
+
+import "io"
+
+// Link is a special kind of value in IPLD which can be "loaded" to access
+// more nodes.
+//
+// Nodes can return a Link; this can be loaded manually, or,
+// the traversal package contains powerful features for automatically
+// traversing links through large trees of nodes.
+//
+// Links straddle somewhat awkwardly across the IPLD Layer Model:
+// clearly not at the Schema layer (though schemas can define their parameters),
+// partially at the Data Model layer (as they're recognizably in the Node interface),
+// and also involved at some serial layer that we don't often talk about:
+// linking -- since we're a content-addressed system at heart -- necessarily
+// involves understanding of concrete serialization details:
+// which encoding mechanisms to use, what string escaping, what hashing, etc,
+// and indeed what concrete serial link representation itself to use.
+//
+// Link is an abstract interface so that we can describe Nodes without
+// getting stuck on specific details of any link representation.
+// In practice, you'll almost certainly use CIDs for linking.
+// However, it's possible to bring your own Link implementations
+// (though this'll almost certainly involve also bringing your own encoding
+// systems; it's a lot of work).
+// It's even possible to use IPLD *entirely without* any linking implementation,
+// using it purely for json/cbor via the encoding packages and
+// foregoing the advanced traversal features around transparent link loading.
+type Link interface {
+	// Load returns a Node identified by the Link.
+	//
+	// The provided Loader function is used to get a reader for the raw
+	// serialized content; the Link contains an understanding of how to
+	// select a decoder (and hasher for verification, etc).
+	Load(loader Loader) (Node, error)
+
+	// LinkBuilder returns a handle to any parameters of the Link which
+	// are needed to create a new Link of the same style but with new content.
+	// (It's much like the relationship of Node/NodeBuilder.)
+	//
+	// (If you're familiar with CIDs, you can think of this method as
+	// corresponding closely to `cid.Prefix()`, just more abstractly.)
+	LinkBuilder() LinkBuilder
+
+	// String should return a reasonably human-readable debug-friendly
+	// representation of a Link.  It should only be used for debug and
+	// log message purposes; there is no contract that requires that the
+	// string be able to be parsed back into a reified Link.
+	String() string
+}
+
+// LinkBuilder encapsulates any implementation details and parameters
+// necessary for taking a Node and converting it to a serial representation
+// and returning a Link to that data.
+//
+// The serialized bytes will be routed through the provided Storer system,
+// which is expected to store them in some way such that a related Loader
+// system can later use the Link and an associated Loader to load nodes
+// of identical content.
+//
+// LinkBuilder, like Link, is an abstract interface.
+// If using CIDs as an implementation, LinkBuilder will encapsulate things
+// like multihashType, multicodecType, and cidVersion, for example.
+type LinkBuilder interface {
+	Build(content Node, storer Storer) (Link, error)
+}
+
+// Loader functions are used to get a reader for raw serialized content
+// based on the lookup information in a Link.
+// A loader function is used by providing it to a Link.Load() call.
+//
+// Loaders typically have some filesystem or database handle contained
+// within their closure which is used to satisfy read operations.
+//
+// LinkContext objects can be provided to give additional information
+// to the loader, and will be automatically filled out when a Loader
+// is used by systems in the traversal package; most Loader implementations
+// should also work fine when given the zero value of LinkContext.
+//
+// Loaders are implicitly coupled to a Link implementation and have some
+// "extra" knowledge of the concrete Link type.  This necessary since there is
+// no mandated standard for how to serially represent Link itself, and such
+// a representation is typically needed by a Storer implementation.
+type Loader func(lnk Link, lnkCtx LinkContext) (io.Reader, error)
+
+// Storer functions are used to a get a writer for raw serialized content,
+// which will be committed to storage indexed by Link.
+// A stoerer function is used by providing it to a LinkBuilder.Build() call.
+//
+// The storer system comes in two parts: the Storer itself *starts* a storage
+// operation (presumably to some e.g. tempfile) and returns a writer; the
+// StoreCommitter returned with the writer is used to *commit* the final storage
+// (much like a 'Close' operation for the writer).
+//
+// Storers typically have some filesystem or database handle contained
+// within their closure which is used to satisfy read operations.
+//
+// LinkContext objects can be provided to give additional information
+// to the storer, and will be automatically filled out when a Storer
+// is used by systems in the traversal package; most Storer implementations
+// should also work fine when given the zero value of LinkContext.
+//
+// Storers are implicitly coupled to a Link implementation and have some
+// "extra" knowledge of the concrete Link type.  This necessary since there is
+// no mandated standard for how to serially represent Link itself, and such
+// a representation is typically needed by a Storer implementation.
+type Storer func(lnkCtx LinkContext) (io.Writer, StoreCommitter, error)
+
+// StoreCommitter is a thunk returned by a Storer which is used to "commit"
+// the storage operation.  It should be called after the associated writer
+// is finished, similar to a 'Close' method, but further takes a Link parameter,
+// which is the identity of the content.  Typically, this will cause an atomic
+// operation in the storage system to move the already-written content into
+// a final place (e.g. rename a tempfile) determined by the Link.
+// (The Link parameter is necessarily only given at the end of the process
+// rather than the beginning to so that we can have content-addressible
+// semantics while also supporting streaming writes.)
+type StoreCommitter func(Link) error
+
+// LinkContext is a parameter to Storer and Loader functions.
+//
+// An example use of LinkContext might be inspecting the LinkNode, and if
+// it's a typed node, inspecting its Type property; then, a Loader might
+// deciding on whether or not we want to load objects of that Type.
+// This might be used to do a traversal which looks at all directory objects,
+// but not file contents, for example.
+type LinkContext struct {
+	LinkPath   Path
+	LinkNode   Node // has the Link again, but also might have type info // always zero for writing new nodes, for obvi reasons.
+	ParentNode Node
+}
+
+// n.b. if I had java, this would all indeed be generic:
+// `Link<$T>`, `LinkBuilder<$T>`, `Storer<$T>`, etc would be an explicit family.
+// ... Then again, in java, that'd prevent composition of a Storer or Loader
+// which could support more than one concrete type, so.  ¯\_(ツ)_/¯