GetRepresentationNodeGen, and impl of struct->map.

This resolves a *lot* of questions that were previously open.
(Progress will probably be faster after this.)

- It's now clear how GetRepresentationNodeGen works at all.
  Turns out it really does just return a nodeGenerator, and that
  works... really well.

- We've got the first example of a 'EmitTypedNodeMethodRepresentation'
  method which generates a switch statement, so that's under the belt.

- Let's not bury the lede: the entire suite of generation code for
  emitting an ipld.Node for the representation of a struct as a map,
  and emitting the entire corresponding ipld.NodeBuilder for building
  a struct out of map entries!  Includes validation for all required
  fields being set, the usual type checks, support for rename mappings,
  and also validation against repeated entries (this lattermost bit is
  a bit controversial, given that there may be other more efficient
  places to do this check, but it's in for now; and see next bullets).

- The solution to the "what if there are multiple possible
  representation implementations?" question is frankly to ignore it.
  I had to think about this a long (long, long) time; time to move on.
  Seealso the comments in the 'EmitNodebuilderMethodCreateMap' method
  on 'generateStructReprMapNb' -- in short, this problem is too big
  to tackle right now.  We also, mostly, *don't need to* -- the
  solution of "push it to the codec layer" can address the correctness
  concerns in all cases I can think of, and the rest is hedging on
  efficiency (for which we really need more complete implementations
  and thereafter *benchmarks* in order to be conclusive anyway).
  Endgame: the current course of action is to build things the way
  that will operate correctly for the widest range of inputs.

- (Note to the future, regarding that last bullet point: some of
  trickiest bits in this choice matrix around efficiency are where
  concerns would be mostly in the codec layer, but would get efficiency
  boosts from knowledge that's only available from the schema layer.
  But the future-planned feature of generating ultra-fastpath direct
  marshal and unmarshal functions with codec specialization will have
  enough information at hand to actually cut straight through all of
  those concerns!)

- Not appearing in this commit, but: expect a fairly huge writeup about
  all these map ordering choices to be coming up in an exploration
  report document in the ipld/specs repo soon.

The two commits coming before this one -- especially the "generality
of codegen helper mixins" one -- also were direct leadups for all this.

Several additional things remain todo:

- This all needs test coverage, and I haven't mustered that far yet.
  Coming in the next commit or so.  I won't be surprised if there's at
  least one bug in this area until those are done.  (I don't like
  committing without tests included, but the current tests probably
  need a small refactor in order to grow smoothly, and I'm not gonna
  try to heap that onto the current diff.  On the plus side: everything
  in the generated output typechecks so far, and that's quite a bit.)

- Support for "implicit" values is missing.  TODOs inline.  They'll
  interact with roughly the same parts of the code as optionals do.

- The representation gen for strings is, as you can see, a todo.
  (It's probably an "easy" one -- but also, it would be nice to get
  it to reuse as much code as possible, because afaict the
  representation node and the type-semantics node are almost identical,
  so that might turn out to be interesting.)

- Note that before we can rig unmarshall up to this and have it work
  recursively and completely, we'll need to address the known todo of
  nodebuilders-need-methods-to-propose-child-nodebuilders.  I've been
  putting that one off for a while, but I think we're coming up on
  when it's correct to get that one done -- just before adding any more
  generators or representations would be good timing.

- Several error paths are still using very stringy errors, and yearn to
  be refactored into typed error structures.  These are mostly the same
  ones as have already appeared in other recent commits; we have
  learned a few more things about which parts of the error message need
  to be flexible, though... so the time to tackle these will also be
  "soon".  (Probably right after we do some more testing work, so we
  can then immediately add tests for the unhappy paths right as we
  upgrade the errors to typed constructions.)

Some other organizational open questions:

- Note that for the type-level node and nodebuilders, we're using two
  separate files; and for the representation and its builder, I haven't
  done so (yet).  Would be good to move to one way or the other.
  Undecided which one is more readable vs shocking yet.

- The names of the types we're using inside the generation isn't very
  consistent right now either.  It's evolving towards consistency as we
  get more cases explored, and I think it's nearly at the mark now, but
  I haven't been proactively refactoring the older stuff yet.  Should;
  but since it'll be roughly sed levels of complexity, not a blocker.

Things that look like tempting todos, but probably aren't:

- It *looks* at first glance like there's a lot of duplicated code
  between the map representation of the struct and the struct itself.
  I'm fairly sure this is a red herring and should not be pursued:
  the places which are the same are many, it's true; but the places
  that are different are wormed in all over the place, and trying to
  extract the common features will likely result in templates which
  are completely unreadable.  This degree of almost-commonality is
  also probably going to be unique in the entire set of kinds and
  representation strategies that we'll deal with, making it further
  unworthy of special attempts at "simplification".  (The strings
  case mentioned above as a todo is different from this, because there,
  things are actually *identical*, not merely close (... I think!).)
  I could be wrong about this, but if so, it'll be better to revisit
  the question after several more kinds and representations get at
  least their first draft.

Whew.  Not sure what the hours-vs-sloc ratio is on this diff, but
it's *high*.  Also worth it: a lot of the future course of development
is set out in the implications of the choices touched on here, and as
much as I'd like to develop iteratively, past experience (on refmt
in particular) tells me some of these will not be easy to revisit.
Signed-off-by: Eric Myhre <hash@exultant.us>

schema/gen/go/gen.go

@@ -43,11 +43,7 @@ type typedNodeGenerator interface {
 	// -- and the representation and its node and nodebuilder -->
 
 	EmitTypedNodeMethodRepresentation(io.Writer)
-	// TODO: EmitRepresentationNode(io.Writer) // deploys *another* whole nodeGenerator
-	// TODO: EmitRepresentationNodeBuilder(io.Writer) // deploys *another* whole nodebuilderGenerator
-
-	// debatable: we could have 'EmitRepresentationNode' and similar return a generator interface instead of just going to work.
-	//  however, this raises questions when it comes to any types which have *multiple* representation-side builders (e.g. strict-order as well as loose-order).
+	GetRepresentationNodeGen() nodeGenerator // includes transitively the matched nodebuilderGenerator
 }
 
 type nodeGenerator interface {

schema/gen/go/genKindString.go

@@ -64,3 +64,7 @@ func (gk generateKindString) EmitTypedNodeMethodRepresentation(w io.Writer) {
 		}
 	`, w, gk)
 }
+
+func (gk generateKindString) GetRepresentationNodeGen() nodeGenerator {
+	return nil // TODO of course
+}

schema/gen/go/genKindStruct.go

@@ -171,8 +171,17 @@ func (gk generateKindStruct) EmitNodeMethodLength(w io.Writer) {
 
 func (gk generateKindStruct) EmitTypedNodeMethodRepresentation(w io.Writer) {
 	doTemplate(`
-		func ({{ .Type.Name }}) Representation() ipld.Node {
-			panic("TODO representation")
+		func (n {{ .Type.Name }}) Representation() ipld.Node {
+			return  _{{ .Type.Name }}__Repr{&n}
 		}
 	`, w, gk)
 }
+
+func (gk generateKindStruct) GetRepresentationNodeGen() nodeGenerator {
+	switch gk.Type.RepresentationStrategy().(type) {
+	case schema.StructRepresentation_Map:
+		return getStructRepresentationMapNodeGen(gk.Type)
+	default:
+		panic("missing case in switch for repr strategy for structs")
+	}
+}

schema/gen/go/genKindStructReprMap.go

+package gengo
+
+import (
+	"io"
+
+	"github.com/ipld/go-ipld-prime/schema"
+)
+
+func getStructRepresentationMapNodeGen(t schema.TypeStruct) nodeGenerator {
+	return generateStructReprMapNode{
+		t,
+		generateKindedRejections_Map{
+			"_" + string(t.Name()) + "__Repr",
+			string(t.Name()) + ".Representation",
+		},
+	}
+}
+
+type generateStructReprMapNode struct {
+	Type schema.TypeStruct
+	generateKindedRejections_Map
+}
+
+func (gk generateStructReprMapNode) EmitNodeType(w io.Writer) {
+	doTemplate(`
+		var _ ipld.Node = _{{ .Type.Name }}__Repr{}
+
+		type _{{ .Type.Name }}__Repr struct{
+			n *{{ .Type.Name }}
+		}
+
+	`, w, gk)
+}
+
+func (gk generateStructReprMapNode) EmitNodeMethodReprKind(w io.Writer) {
+	doTemplate(`
+		func (_{{ .Type.Name }}__Repr) ReprKind() ipld.ReprKind {
+			return ipld.ReprKind_Map
+		}
+	`, w, gk)
+}
+
+func (gk generateStructReprMapNode) EmitNodeMethodLookupString(w io.Writer) {
+	// almost idential to the type-level one, just with different strings in the switch.
+	// TODO : support for implicits is missing.
+	doTemplate(`
+		func (rn _{{ .Type.Name }}__Repr) LookupString(key string) (ipld.Node, error) {
+			switch key {
+			{{- $type := .Type -}} {{- /* ranging modifies dot, unhelpfully */ -}}
+			{{- range $field := .Type.Fields }}
+			case "{{ $field | $type.RepresentationStrategy.GetFieldKey }}":
+				{{- if and $field.IsOptional $field.IsNullable }}
+				if !rn.n.{{ $field.Name }}__exists {
+					return ipld.Undef, nil
+				}
+				if rn.n.{{ $field.Name }} == nil {
+					return ipld.Null, nil
+				}
+				{{- else if $field.IsOptional }}
+				if rn.n.{{ $field.Name }} == nil {
+					return ipld.Undef, nil
+				}
+				{{- else if $field.IsNullable }}
+				if rn.n.{{ $field.Name }} == nil {
+					return ipld.Null, nil
+				}
+				{{- end}}
+				{{- if or $field.IsOptional $field.IsNullable }}
+				return *rn.n.{{ $field.Name }}, nil
+				{{- else}}
+				return rn.n.{{ $field.Name }}, nil
+				{{- end}}
+			{{- end}}
+			default:
+				return nil, typed.ErrNoSuchField{Type: nil /*TODO*/, FieldName: key}
+			}
+		}
+	`, w, gk)
+}
+
+func (gk generateStructReprMapNode) EmitNodeMethodLookup(w io.Writer) {
+	doTemplate(`
+		func (rn _{{ .Type.Name }}__Repr) Lookup(key ipld.Node) (ipld.Node, error) {
+			ks, err := key.AsString()
+			if err != nil {
+				return nil, ipld.ErrInvalidKey{"got " + key.ReprKind().String() + ", need string"}
+			}
+			return rn.LookupString(ks)
+		}
+	`, w, gk)
+}
+
+func (gk generateStructReprMapNode) EmitNodeMethodMapIterator(w io.Writer) {
+	// Amusing note, the iterator ends up with a loop in its body, even though
+	//  it only yields one entry pair at a time -- this is needed so we can
+	//   use 'continue' statements to skip past optionals which are undefined.
+	// TODO : support for implicits is missing.
+	doTemplate(`
+		func (rn _{{ .Type.Name }}__Repr) MapIterator() ipld.MapIterator {
+			return &_{{ .Type.Name }}__itr{rn.n, 0}
+		}
+
+		type _{{ .Type.Name }}__ReprItr struct {
+			node *{{ .Type.Name }}
+			idx  int
+		}
+
+		func (itr *_{{ .Type.Name }}__ReprItr) Next() (k ipld.Node, v ipld.Node, _ error) {
+			if itr.idx >= {{ len .Type.Fields }} {
+				return nil, nil, ipld.ErrIteratorOverread{}
+			}
+			for {
+				switch itr.idx {
+				{{- $type := .Type -}} {{- /* ranging modifies dot, unhelpfully */ -}}
+				{{- range $i, $field := .Type.Fields }}
+				case {{ $i }}:
+					k = String{"{{ $field | $type.RepresentationStrategy.GetFieldKey }}"}
+					{{- if and $field.IsOptional $field.IsNullable }}
+					if !itr.node.{{ $field.Name }}__exists {
+						itr.idx++
+						continue
+					}
+					if itr.node.{{ $field.Name }} == nil {
+						v = ipld.Null
+						break
+					}
+					{{- else if $field.IsOptional }}
+					if itr.node.{{ $field.Name }} == nil {
+						itr.idx++
+						continue
+					}
+					{{- else if $field.IsNullable }}
+					if itr.node.{{ $field.Name }} == nil {
+						v = ipld.Null
+						break
+					}
+					{{- end}}
+					v = itr.node.{{ $field.Name }}
+				{{- end}}
+				default:
+					panic("unreachable")
+				}
+			}
+			itr.idx++
+			return
+		}
+		func (itr *_{{ .Type.Name }}__ReprItr) Done() bool {
+			return itr.idx >= {{ len .Type.Fields }}
+		}
+
+	`, w, gk)
+
+}
+
+func (gk generateStructReprMapNode) EmitNodeMethodLength(w io.Writer) {
+	// This is fun: it has to count down for any unset optional fields.
+	// TODO : support for implicits is missing.
+	doTemplate(`
+		func (rn _{{ .Type.Name }}__Repr) Length() int {
+			l := {{ len .Type.Fields }}
+			{{- range $field := .Type.Fields }}
+			{{- if and $field.IsOptional $field.IsNullable }}
+			if !rn.n.{{ $field.Name }}__exists {
+				l--
+			}
+			{{- else if $field.IsOptional }}
+			if rn.n.{{ $field.Name }} == nil {
+				l--
+			}
+			{{- end}}
+			{{- end}}
+			return l
+		}
+	`, w, gk)
+}
+
+func (gk generateStructReprMapNode) EmitNodeMethodNodeBuilder(w io.Writer) {
+	doTemplate(`
+		func (_{{ .Type.Name }}__Repr) NodeBuilder() ipld.NodeBuilder {
+			return _{{ .Type.Name }}__ReprBuilder{}
+		}
+	`, w, gk)
+}
+
+func (gk generateStructReprMapNode) GetNodeBuilderGen() nodebuilderGenerator {
+	return generateStructReprMapNb{
+		gk.Type,
+		genKindedNbRejections_Map{
+			"_" + string(gk.Type.Name()) + "__ReprBuilder",
+			string(gk.Type.Name()) + ".Representation.Builder",
+		},
+	}
+}
+
+type generateStructReprMapNb struct {
+	Type schema.TypeStruct
+	genKindedNbRejections_Map
+}
+
+func (gk generateStructReprMapNb) EmitNodebuilderType(w io.Writer) {
+	// Note there's no need to put the reprKind in the name of the type
+	//  we generate here: there's only one representation per type.
+	//   (We *could* munge the reprkind in for debug symbol reading,
+	//    but at present it hasn't seemed warranted.)
+	doTemplate(`
+		type _{{ .Type.Name }}__ReprBuilder struct{}
+
+	`, w, gk)
+}
+
+func (gk generateStructReprMapNb) EmitNodebuilderMethodCreateMap(w io.Writer) {
+	// Much of these looks the same as the type-level builders.  Key differences:
+	//  - We interact with the rename directives here.
+	//  - The "__isset" bools are generated for *all* fields -- we need these
+	//     to check if a key is repeated, so we can reject that.
+	//      Worth mentioning: we could also choose *not* to check this, instead
+	//       insisting it's a codec layer concern.  This needs revisiting;
+	//        at present I'm choosing "defense in depth", because trying to
+	//         reason out the perf and usability implications in advance has
+	//          yielded a huge matrix of concerns and no single clear gradient.
+	// TODO : support for implicits is missing.
+	doTemplate(`
+		func (nb _{{ .Type.Name }}__ReprBuilder) CreateMap() (ipld.MapBuilder, error) {
+			return &_{{ .Type.Name }}__ReprMapBuilder{v:&{{ .Type.Name }}{}}, nil
+		}
+
+		type _{{ .Type.Name }}__ReprMapBuilder struct{
+			v *{{ .Type.Name }}
+			{{- range $field := .Type.Fields }}
+			{{ $field.Name }}__isset bool
+			{{- end}}
+		}
+
+		func (mb *_{{ .Type.Name }}__ReprMapBuilder) Insert(k, v ipld.Node) error {
+			ks, err := k.AsString()
+			if err != nil {
+				return ipld.ErrInvalidKey{"not a string: " + err.Error()}
+			}
+			switch ks {
+			{{- $type := .Type -}} {{- /* ranging modifies dot, unhelpfully */ -}}
+			{{- range $field := .Type.Fields }}
+			case "{{ $field | $type.RepresentationStrategy.GetFieldKey }}":
+				if mb.{{ $field.Name }}__isset {
+					panic("repeated assignment to field") // FIXME need an error type for this
+				}
+				{{- if $field.IsNullable }}
+				if v.IsNull() {
+					mb.v.{{ $field.Name }} = nil
+					{{- if $field.IsOptional }}
+					mb.v.{{ $field.Name }}__exists = true
+					{{- else}}
+					mb.{{ $field.Name }}__isset = true
+					{{- end}}
+					return nil
+				}
+				{{- else}}
+				if v.IsNull() {
+					panic("type mismatch on struct field assignment: cannot assign null to non-nullable field") // FIXME need an error type for this
+				}
+				{{- end}}
+				tv, ok := v.(typed.Node)
+				if !ok {
+					panic("need typed.Node for insertion into struct") // FIXME need an error type for this
+				}
+				x, ok := v.({{ $field.Type.Name }})
+				if !ok {
+					panic("field '{{$field.Name}}' (key: '{{ $field | $type.RepresentationStrategy.GetFieldKey }}') in type {{$type.Name}} is type {{$field.Type.Name}}; cannot assign "+tv.Type().Name()) // FIXME need an error type for this
+				}
+
+				{{- if or $field.IsOptional $field.IsNullable }}
+				mb.v.{{ $field.Name }} = &x
+				{{- else}}
+				mb.v.{{ $field.Name }} = x
+				{{- end}}
+				{{- if and $field.IsOptional $field.IsNullable }}
+				mb.v.{{ $field.Name }}__exists = true
+				{{- end}}
+				mb.{{ $field.Name }}__isset = true
+			{{- end}}
+			default:
+				return typed.ErrNoSuchField{Type: nil /*TODO:typelit*/, FieldName: ks}
+			}
+			return nil
+		}
+		func (mb *_{{ .Type.Name }}__ReprMapBuilder) Delete(k ipld.Node) error {
+			panic("TODO later")
+		}
+		func (mb *_{{ .Type.Name }}__ReprMapBuilder) Build() (ipld.Node, error) {
+			{{- $type := .Type -}} {{- /* ranging modifies dot, unhelpfully */ -}}
+			{{- range $field := .Type.Fields }}
+			{{- if not $field.IsOptional }}
+			if !mb.{{ $field.Name }}__isset {
+				panic("missing required field '{{$field.Name}}' (key: '{{ $field | $type.RepresentationStrategy.GetFieldKey }}') in building struct {{ $type.Name }}") // FIXME need an error type for this
+			}
+			{{- end}}
+			{{- end}}
+			v := mb.v
+			mb = nil
+			return v, nil
+		}
+
+	`, w, gk)
+}
+
+func (gk generateStructReprMapNb) EmitNodebuilderMethodAmendMap(w io.Writer) {
+	doTemplate(`
+		func (nb _{{ .Type.Name }}__ReprBuilder) AmendMap() (ipld.MapBuilder, error) {
+			panic("TODO later")
+		}
+	`, w, gk)
+}

schema/gen/go/gen_test.go

@@ -36,9 +36,8 @@ func TestNuevo(t *testing.T) {
 		tg.EmitNodeMethodAsString(w)
 		tg.EmitNodeMethodAsBytes(w)
 		tg.EmitNodeMethodAsLink(w)
-		tg.EmitNodeMethodNodeBuilder(w)
-		tg.EmitTypedNodeMethodRepresentation(w)
 
+		tg.EmitNodeMethodNodeBuilder(w)
 		tnbg := tg.GetNodeBuilderGen()
 		tnbg.EmitNodebuilderType(w)
 		tnbg.EmitNodebuilderMethodCreateMap(w)
@@ -52,6 +51,43 @@ func TestNuevo(t *testing.T) {
 		tnbg.EmitNodebuilderMethodCreateString(w)
 		tnbg.EmitNodebuilderMethodCreateBytes(w)
 		tnbg.EmitNodebuilderMethodCreateLink(w)
+
+		tg.EmitTypedNodeMethodRepresentation(w)
+		rng := tg.GetRepresentationNodeGen()
+		if rng == nil { // FIXME: hack to save me from stubbing tons right now, remove when done
+			return
+		}
+		rng.EmitNodeType(w)
+		rng.EmitNodeMethodReprKind(w)
+		rng.EmitNodeMethodLookupString(w)
+		rng.EmitNodeMethodLookup(w)
+		rng.EmitNodeMethodLookupIndex(w)
+		rng.EmitNodeMethodMapIterator(w)
+		rng.EmitNodeMethodListIterator(w)
+		rng.EmitNodeMethodLength(w)
+		rng.EmitNodeMethodIsUndefined(w)
+		rng.EmitNodeMethodIsNull(w)
+		rng.EmitNodeMethodAsBool(w)
+		rng.EmitNodeMethodAsInt(w)
+		rng.EmitNodeMethodAsFloat(w)
+		rng.EmitNodeMethodAsString(w)
+		rng.EmitNodeMethodAsBytes(w)
+		rng.EmitNodeMethodAsLink(w)
+
+		rng.EmitNodeMethodNodeBuilder(w)
+		rnbg := rng.GetNodeBuilderGen()
+		rnbg.EmitNodebuilderType(w)
+		rnbg.EmitNodebuilderMethodCreateMap(w)
+		rnbg.EmitNodebuilderMethodAmendMap(w)
+		rnbg.EmitNodebuilderMethodCreateList(w)
+		rnbg.EmitNodebuilderMethodAmendList(w)
+		rnbg.EmitNodebuilderMethodCreateNull(w)
+		rnbg.EmitNodebuilderMethodCreateBool(w)
+		rnbg.EmitNodebuilderMethodCreateInt(w)
+		rnbg.EmitNodebuilderMethodCreateFloat(w)
+		rnbg.EmitNodebuilderMethodCreateString(w)
+		rnbg.EmitNodebuilderMethodCreateBytes(w)
+		rnbg.EmitNodebuilderMethodCreateLink(w)
 	}
 
 	f := openOrPanic("_test/minima.go")